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INTRODUCTION 


^A  major  function  of  human  factors  engineering  throughout  the  system 
development  process  is  to  ensure  that  system  demands  do  not  exceed  the  information 
processing  capabilities  of  the  human  operator.  Processing  overload  is  a  central 
factor  leading  to  breakdowns  in  operator  performance  and  to  the  compromises  in 
system  safety  and  effectiveness  that  can  result  from  such  decrements.  Mental 
workload  is  the  term  which  refers  to  that  portion  of  an  operator’s  limited  pro¬ 
cessing  capacity  which  is  actually  required  to  perform  a  particular  task  or  system 
function.  The  principal  objective  of  workload  assessment  is  to  specify  the  amount 
of  expanded  processing  capacity  so  that  existing  or  potential  overloads  can  be 
identified  and  decrements  in  operator  performance  avoided.  / 


aea.  < 
nwSlog 


The  use  of  advanced  display  and  control  technologies  in  modern  weapons 
systems  has  been  accompanied  in  many  instances  by  substantial  increases  in  the 
monitoring,  supervisory,  and  decision-making  demands  imposed  on  the  operator. 
These  heavy  demands  have  markedly  increased  the  likelihood  of  approaching  or 
actually  exceeding  operator  processing  capacity  limits.  As  a  consequence,  assess¬ 
ment  of  the  mental  workload  imposed  by  alternative  design  options  has  become  par¬ 
ticularly  critical  throughout  the  weapon  system  design  process. 

Because  of  its  critical  role  in  the  system  development  process,  workload 
assessment  has  been  the  subject  of  considerable  research  over  the  past  10  years 
(e.g.,  Moray,  1979).  One  product  of  these  research  efforts  has  been  the  develop¬ 
ment  and  application  of  a  large  number  of  individual  workload  assessment  tech¬ 
niques.  A  recent  comprehensive  review  (Wierwille  .and  Williges,  1978)  of  the 
workload  assessment  literature,  for  example,  identified  28  different  techniques 
that  had  been  used  to  derive  measures  of  load.  A  substantial  number  of  these 
empirical  assessment  techniques  can  be  classified  as  belonging  to  one  of  three 
categories  of  workload  measures:  (1)  subjective  opinion  procedures, 

(2)  performance-based  techniques,  and  (3)  physiological  techniques.^-, _ 


Subjective  techniques  (e.g.,  Gartner  and  Murphy,  1976;  Williges  and 
Wierwille,  1979;  Moray,  1982)  require  that  the  operator  judge  and  report  the 
degree  cf  workload  experienced  during  performance  of  a  particular  task  or  system 


function.  Rating  scales  are  the  most  frequently  used  type  of  subjective  measure- 
menc  technique. 


Performance-based  techniques  (e.g.,  Gartner  and  Murphy,  1976;  Williges  and 
Wierwille,  1979)  use  some  measure  of  operator  behavior  or  activity  as  the  basis  of 
a  workload  index.  A  number  of  individual  assessment  techniques  can  be  categorized 
as  performance-based  measures.  So-called  primary  task  techniques  (e.g.,  Rolfe, 
1976;  Gartner  and  Murphy,  1976;  Williges  and  Wierwille,  1979)  examine  some  aspect 
of  the  operator's  capability  to  perform  the  task  or  system  function  of  interest  in 
order  to  provide  an  estimate  of  load.  Deviations  from  glideslope  by  a  pilot  on 
final  approach  would  constitute  one  such  primary  task  measure.  A  second  type  of 
performance-based  measure  which  has  been  frequently  used  to  assess  workload  is 
secondary  task  methodology  (e.g.,  Knowles,  1963;  Rolfe,  1971;  Ogden,  Levine,  and 
Eisner,  1979;  Williges  and  Wierwille,  1979).  This  approach  derives  an  estimate  of 
workload  from  the  operator's  capability  to  perform  a  secondary  task  concurrently 
with  the  primary  task  of  interest. 

Physiological  techniques  (e.g.,  O'Donnell,  1979;  Wierwille,  1979)  measure 
some  aspect  of  the  operator's  physiological  response  to  task  or  system  demand,  and 
provide  a  measure  of  load  based  on  these  responses.  A  wide  variety  of  physio¬ 
logical  measures  (e.g.,  heart  rate  variability,  pupil  diameter,  event-related 
brain  potentials)  have  been  used  in  order  to  assess  workload. 


Since  a  variety  of  workload  assessment  procedures  are  available,  an  important 
decision  faced  by  a  system  designer  involves  choice  of  the  technique  that  best 
meets  design  requirements.  The  system  development  process  typically  involves  a 
series  of  stages  which  range  from  conceptual  development  through  operational  test 
and  evaluation  of  the  system.  These  stages  can  be  characterized  by  variations  in 
both  the  specific  questions  addressed  by  workload  measurement,  and  in  the  prac¬ 
tical  constraints  that  must  be  satisfied  by  assessment  techniques.  These  ques¬ 
tions  3nd  constraints  suggest  a  number  of  criteria  that  should  be  considered  in 
choosing  a  workload  measure  for  application  during  system  development.  The  pur¬ 
poses  of  this  paper  are  to  outline  a  set  of  such  criteria,  briefly  review  the 
current  status  of  the  three  classes  of  empirical  techniques  as  they  relate  to  the 
proposed  criteria,  and  suggest  some  applications  for  each  class  of  technique 
during  system  development.  Some  recent  work  with  a  subjective  assessment  pro¬ 
cedure  which  has  the  potential  for  application  throughout  the  system  development 
process  is  also  discussed. 


WORKLOAD  METRIC  SELECTION  CRITERIA 

A  number  of  criteria  for  evaluation  of  workload  metrics  have  been  proposed  in 
the  recent  literature  (e.g.,  Gartner  and  Murphy,  1976;  Rolfe,  1976;  Ogden  et  al., 
1979;  Williges  and  Wierwille,  1979;  Wickens,  1981;  Shingledecker ,  1983).  Several 
of  the  proposed  criteria  are  particularly  relevant  for  choice  of  a  metric  during 
system  design.  These  criteria  include:  (1)  sensitivity,  (2)  dlagnosticity, 
(3)  intrusiveness,  (4)  implementationn  requirements,  and  (5)  operator  acceptance. 


Sens  it ivi ty 

Sensitivity  refers  to  the  capability  of  a  measure  to  distinguish  different 
levels  of  load  imposed  by  a  task  or  design  option.  The  degree  of  sensitivity 
required  in  an  assessment  technique  is  directly  related  to  the  nature  of  the  ques¬ 
tion  to  be  answered  by  the  workload  measure.  There  are  a  wide  variety  of  specific 
design  questions  (e.g.,  adequacy  of  control/display  design,  allocation 
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functions  between  operators)  that  can  be  addressed  by  workload  assessment  during 
system  development.  Regardless  of  the  specific  aspect  of  the  design  that  is 
addressed,  however,  the  two  basic  objectives  of  workload  assessment  are  to  deter¬ 
mine:  (1)  if  an  overload  that  would  lead  to  degraded  operator  performance 

actually  exists,  or  (2)  if  the  potential  for  such  an  overload  exists.  Questions 
involving  the  first  objective  can  be  addressed  through  primary  task  performance 
measures,  since  they  are  generally  assumed  to  differentiate  overload  from  nonover¬ 
load  situations  (e.g.,  Knowles,  1963;  Gartner  and  Murphy,  1976;  Williges  and 
Wierwille,  1979).  In  other  applications,  however,  a  designer  might  wish  to 
evaluate  the  potential  for  overload  among  several  design  options  that  yield 
adequate  operator  performance.  This  objective  is  relevant  when  it  is  anticipated 
that  other  factors  during  system  operation  Ce.g. ,  environmental  stressors,  equip¬ 
ment  failures)  might  contribute  additional  load  chat  would  be  sufficient  to  cause 
degraded  operator  performance.  In  this  instance,  even  though  none  of  the  design 
options  themselves  overload  the  operator,  it  is  desirable  to  identify  the  option 
that  Imposes  the  lowest  load  and  affords  the  greatest  reserve  capacity  for  dealing 
with  other  sources  of  demand.  This  type  of  evaluation  would  require  a  workload 
measure  that  was  more  sensitive  to  variations  in  load  than  primary  task  measures, 
and  would  suggest  the  use  of  other  procedures  (e.g.,  subjective,  physiological, 
secondary  task)  that  are  designed  to  discriminate  levels  of  workload  in  nonover¬ 
load  situations.  Current  evidence  indicates,  for  example,  that  both  secondary 
task  measures  (e.g.,  Schifflet,  Linton,  and  Spicuzza,  1982)  and  subjective  ratings 
of  load  (e.g.,  Eggemeier,  Crabtree,  and  LaPointe,  1983)  can  discriminate  differ¬ 
ences  in  task  demand  that  are  not  reflected  in  primary  task  measures  of  operator 
performance.  The  sensitivity  criterion  is,  therefore,  an  essential  consideration 
in  choice  of  a  workload  measure,  since  the  degree  of  sensitivity  bears  directly  on 
the  type  of  question  that  can  be  addressed  by  a  technique. 


Diagnostic! ty 

Diagnosticity  (Wickens,  1981;  Wickens  and  Derrick,  1981;  Shingledecker,  1983) 
is  a  second  important  consideration  in  choice  of  a  system  evaluation  metric.  This 
criterion  is  based  on  the  multiple  resources  theory  (e.g.,  Navon  and  Gopher*  1979; 
Sanders,  1979;  Wickens,  1981)  explanation  of  limitations  within  the  human  pro¬ 
cessing  system.  Essentially,  this  theory  holds  that  the  processing  capacity 
expended  in  task  performance  is  not  unitary,  but  is  drawn  from  multiple  sources  or 
pools,  each  with  its  own  resources  that  cannot  be  exchanged  with  other  pools.  One 
version  of  multiple  resources  theory  (Wickens,  1981)  maintains  that  perceptual  and 
central  processing  stages  within  the  human  system  draw  on  one  resource  pool,  while 
the  response  or  motor  output  stage  draws  from  a  separate  resource  pool.  Under 
this  position,  it  is  possible  to  overload  or  fully  expend  thq  resources  associated 
with  one  source,  while  not  depleting  the  processing  resources  of  another  source. 
For  example,  the  requirement  to  monitor  a  display  which  places  heavy  demands  on 
short-term  memory  might  overload  perceptual/central  processing  resources,  while 
making  minimal  demands  on  motor  output  resources.  Other  system  requirements  such 
as  a  final  approach  in  an  aircraft  would  have  a  different  demand  composition,  and 
might  require  greater  expenditures  of  motor  output  resources.  Diagnosticity 
refers  to  the  capability  of  a  technique  to  discriminate  these  differences  in  the 
load  imposed  on  specific  operator  resources. 

It  has  been  proposed  (Wickens,  1981;  Wickens  and  Derrick,  1981)  that  workload 
measures  vary  in  their  degree  of  diagnosticity.  There  are  data  which  indicate, 
for  example,  that  some  physiological  measures  such  as  pupil  diameter  (e.g.,  Beatty 
and  Kahneman,  1966;  Jiang  and  Beatty,  1981)  and  some  subjective  rating  scales 
(e.g.,  Reid,  Shingled*  <-  ker ,  and  Eggemeier,  1981a;  Eggemeier,  Crabtree,  Zingg, 
Reid,  and  Shingledecker.  1982;  Notestine,  1983;  Wierwille  and  Casali,  1983a)  are 
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sensitive  to  perceptuaL,  central  processing,  and  response  load  manipulations.  The 
event-related  brain  potential  (lsreal,  Chesney,  Wickens,  and  Donchin,  1980; 
Isreal,  ,  Wickens,  Chesney,  Donchin,  1980)  and  some  secondary  tasks  (e.g..  North, 

1977;  Wickens  and  Kessel,  1980;  Shingledecker ,  Acton,  and  Crabtree,  1983),  how- 

ever,  show  differential  sensitivity  to  manipulations  of  perceptual/central  pro¬ 

cessing  and  motor  output  demands.  These  data  imply  that  subjective  rating  scales 
and  some  physiological  measures  are  not  particularly  diagnostic,  and  can  prove 

sensitive  to  variations  in  resource  expenditure  anywhere  within  the  human  pro¬ 
cessing  system.  However,  other  physiological  metrics  and  various  secondary  tasks 
appear  to  be  more  diagnostic  of  specific  types  of  resource  or  capacity 

expenditure. 

Such  differences  in  diagnosticity  suggest  that  the  different  types  of 
measures  can  play  complementary  roles  during  system  development.  Less  diagnostic 
measures  could  serve  as  screening  devices  to  initially  determine  if  high  levels  of 
loading  exist  during  performance  of  a  task  or  system  function,  while  more  diagnos¬ 
tic  procedures  could  be  subsequently  used  to  pinpoint  the  particular  source  (e.g., 
perceptual  versus  motor  output)  of  any  such  overloads.  Choice  of  an  assessrent 
technique  on  the  basis  of  the  diagnosticity  criterion  would,  therefore,  be  depend¬ 
ent  on  the  objective  to  be  met  by  the  measure  of  workload. 

Intrusiveness 

While  the  criteria  of  sensitivity  and  diagnosticity  relate  to  the  nature  of 
the  question  that  is  to  be  addressed  by  a  workload  measure,  there  are  a  number  of 
additional  criteria  that  are  suggested  by  practical  constraints  imposed  on  the  use 
of  metrics  during  the  system  development  process.  The  characteristic  of  intru- 
siveness  (e.g.,  Gartner  and  Murphy,  1976;  Wllliges  and  Wierville,  1979;  ^ 

Shingledecker,  1983)  is  one  such  criterion,  and  refers  to  the  tendency  for  some  "  ' 
metrics  to  cause  degradations  in  ongoing  primary  task  performance. 

Intrusiveness  in  an  assessment  procedure  is  unuersirable  on  both  practical 
and  theoretical  grounds.  From  a  practical  perspective,  it  is  clear  that  any  tech¬ 
nique  that  causes  decrements  in  operator  performance  can  potentially  compromise 
the  safety  of  system  operation.  Such  compromises  are  obviously  unacceptable, 
particularly  during  the  later  stages  of  system  development  when  operational  test 
and  evaluations  of  prototype  or  initial  production  models  are  conducted.  From  a 
theoretical  point  of  view,  intrusiveness  can  cause  problems  in  the  interpretation 
of  data  resulting  from  application  of  an  assessment  technique.  These  interpreta¬ 
tion  problems  stem  from  the  assumption  that  measurement  procedures  provide  a  pure 
index  of  the  load  imposed  by  the  primary  task.  If  primary  task  performance  is 
degraded  by  the  introduction  of  the  assessment  technique,  an  unbiased  measure  of 
primary  task  workload  is  not  possible.  Although  intrusiveness  presents  potential 
difficulties  for  all  metrics,  the  interpretation  problem  can  be  particularly  acute 
with  secondary  task  measures  (Rolfe,  1971;  Ogden  et  al.,  1979)  that  are  intended 
to  provide  a  measure  of  the  reserve  capacity  afforded  by  the  primary  task. 

Despite  its  importance,  the  comparative  data  base  on  the  degree  of  intrusion 
associated  with  the  various  types  of  metrics  is  not  extensive.  Some  significant, 
steps  toward  establishing  a  systematic  data  base  have  been  undertaken  recently 
(e.g.,  Casali  and  Wierwille,  1982,  1983;  Rahimi  and  Wierville,  1982; 

Shingledecker,  Crabtree,  and  Acton,  1982;  Acton,  Crabtree,  and  Shingledecker, 

1983;  Wierwille  and  Casali,  1983b;  Wierville  and  Conner,  1983),  but  such  direct  ^ 
comparison  data  are  not  yet  complete.  However,  some  statements  regarding  the 
potential  for  intrusiveness  can  be  made  on  the  basis  of  data  generated  by  indi¬ 
vidual  applications  of  the  various  techniques. 
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First,  it  is  clear  that  Intrusiveness  has  represented  a  major  problem  in  many 
applications  of  secondary  task  methodology  (e.g.  ,  Rolfe,  1971;  Gartner  and  Murphy, 
1976;  Ogden  et  al. ,  1979;  Wllliges  and  Wlerwille,  1979).  The  problem  has  led  to 
the  development  of  techniques  such  as  cross-adaptive  (e.g.  ,  Kelly  and  Wargo,  1967; 
Jex  and  Clement,  1979)  and  embedded  (Shingledecker,  Crabtree,  Simons,  Courtright, 
and  O'Donnell,  1980;  Shingledecker,  1980;  Crabtree  and  Spicuzza,  1981; 
Shingledecker  and  Crabtree,  1982)  secondary  tasks  that  are  designed  to  minimize  or 
control  the  levels  of  intrusion.  Cross-adaptive  procedures  permit  variations  in 
secondary  task  difficulty  as  a  function  of  primary  task  performance.  When  primary 
task  performance  falls  below  a  specified  criterion,  secondary  task  difficulty  is 
reduced  in  order  to  control  the  level  of  intrusion.  This  type  of  procedure  has 
been  successfully  employed  in  a  number  of  laboratory  and  simulation  studies  (Kelly 
and  Wargo,  1967;  Jex  and  Clement,  1979)  that  have  utilized  primary  continuous 
cracking  casks.  Applications  of  the  procedure  to  discrete  tasks  in  more  complex 
environments  have  not  been  accomplished,  and  could  present  difficulties  due  to 
problems  in  obtaining  primary  task  measures  Chat  would  permit  adaptation  of  the 
secondary  task.  The  embedded  secondary  task  approach,  on  the  other  hand,  was 
developed  for  application  to  high  fidelity  simulation  or  operational  environ¬ 
ments.  This  procedure  uses  an  element  already  embedded  in  normal  system  operation 
procedures  as  the  secondary  task.  The  elements  chosen  as  secondary  tasks  (e.g., 
radio  communications)  are  those  that  are  normally  assigned  lower  priority  than  the 
primary  task  (e.g.,  flight  control),  thereby  minimizing  the  potential  for  primary 
cask  intrusion. 

Second,  it  appears  that  the  intrusion  associated  with  most  other  classes  of 
assessment  techniques  tends  to  be  minimal.  Subjective  assessment  techniques 
typically  present  no  significant  intrusion  problem,  6ir  e  rating  scales  and  other 
report  procedures  are  usually  completed  subsequent  to  primary  task  performance. 
Primary  task  measures  are,  by  definition,  nonintrusive,  because  their  application 
involves  no  additional  operator  performance  or  reports.  Physiological  procedures 
also  appear  to  minimize  the  potential  for  intrusion,  although  there  are  data 
(Rahirai  and  Wlerwille,  1982)  which  indicate  that  these  techniques  can  be  associ¬ 
ated  with  some  intrusion. 

The  degree  of  intrusiveness  that  can  be  tolerated  in  an  assessment  technique 
will  vary  as  a  function  of  the  context  in  which  the  measure  is  taken.  Some  degree 
of  intrusion  in  a  simulator  or  in  a  crewstation  mockup  could  be  less  serious,  for 
example,  than  equivalent  levels  of  primary  task  decrement  during  actual  system 
operation.  Choice  of  an  assessment  procedure  on  the  basis  of  intrusiveness  would, 
therefore,  be  determined  in  part  by  constraints  dictated  by  the  measurement 
situation. 

Implementation  Requirements 

The  Implementation  requirements  associated  with  a  particular  measurement 
technique  constitute  a  second  criterion  that  is  heavily  influenced  by  the  prac¬ 
tical  constraints  imposed  by  the  system  development  process.  Implementation 
requirements  are  factors  that  are  related  to  the  ease  with  which  a  technique  can 
be  applied  at  different  stages  of  system  {levelopment  and  evaluation.  Examples  of 
such  factors  include:  (1)  the  Instrumentation  and  software  that  is  required  to 

record  and  analyze  the  measures  associated  with  a  technique;  (2)  any  operator 
training  that  is  necessary  for  the  technique  to  be  properly  applied;  and  (3)  sys¬ 
tem  simulation  facilities  or  actual  equipment  that  are  required  for  application  of 
the  technique. 
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Different  classes  of  assessment  procedures  can  vary  considerably  In  their 
Instrumentation  requirements,  as  can  Individual  techniques  within  the  same  cate¬ 
gory.  ■  For  instance,  subjective  opinion  measures  usually  make  use  of  paper  and.vJS* 
pencil  for  data  recording,  while  much  more  stringent  implementation  requirements 
are  typically  associated  with  physiological  and  some  performance-based  proce¬ 
dures.  Requirements  also  vary  within  categories  themselves.  Cross-adaptive 
secondary  techniques  require  more  extensive  instrumentation  than  other  secondary 
task  procedures  (e.g. ,  interval  production,  Shingledecker  et  al.,  1983)  which 
require  only  a  means  of  recording  an  operator's  response.  Therefore,  when  minimal 
instrumentation  is  a  primary  constraint,  the  use  of  subjective  measures  or  certain 
secondary  task  procedures  such  as  the  interval  production  task  is  suggested. 

Operator  training  requirements  also  vary  with  techniques  and  can  be  necessary 
with  both  secondary  task  and  subjective  assessment  procedures.  Applications  of 
secondary  task  methodology,  for  instance,  usually  require  some  operator  training 
in  order  to  stabilize  baseline  performance  on  the  secondary  task  before  it  is 
performed  concurrently  with  the  primary  task.  Some  subjective  procedures  (e.g., 

Reid  et  al. ,  1981a)  also  Include  the  provision  for  familiarization  with  the  rating 
scales  prior  to  their  use.  Training  requirements  associated  with  the  use  of  both 
primacy  task  and  physiological  measures  would  be  virtually  nonexistent  in  most 
cases. 

Techniques  can  also  differ  in  the  types  of  simulation  facilities  and 
operational  equipment  that  are  necessary  for  their  application.  Such  facility 
requirements  can  be  particularly  restrictive  during  the  early  conceptual  stage  of 
system  development,  when  system  design  information  is  very  general,  and  simulation 
and  raockup  facilities  are  typically  not  available.  Since  both  performance-based 
and  physiological  techniques  require  such  facilities,  their  application  has  been  * 
usually  restricted  to  later  stages  (e.g.,  validation,  engineering  development)  of?.., 
the  design  process  when  the  appropriate  devices  are  present.  This  constraint  on 
early  use  of  physiological  and  performance-based  procedures  is  one  factor  that  has 
led  to  the  development  and  application  of  analytical  time-line  techniques  (e.g., 
Zipoy,  Premselaar,  Gargett,  Belyea,  and  Hall,  1970;  Parks,  1979;  Geer,  1981)  and 
several  simulation  models  (e.g.,  Linton,  Jahns,  and  Chatelier,  1977;  Lane,  Strieb, 
and  Wherry,  1977;  Lane,  Strieb,  Glenn,  and  Wherry,  1981;  Chubb,  1981)  that  are 
capable  of  addressing  workload  assessment  issues  during  earlier  stages  of 
design.  Traditional  applications  of  subjective  metrics  also  require  the  avail¬ 
ability  of  mockups,  simulators,  or  operational  equipment.  However,  a  recent 
application  (Quinn,  Jauer,  and  Summers,  1982)  demonstrated  the  projective  use  of  a 
subjective  metric  by  requiring  experienced  pilots  to  rate  the  expected  load 
associated  with  several  proposed  cockpit  enhancements.  The  projective  ratings 
were  based  on  detailed  descriptions  of  mission  profiles  and  control/display 
options,  and  were  intended  to  provide  workload  estimates  that  could  be  combined 
with  other  factors  (e.g.,  cost)  to  initially  screen  design  options  for  further 
evaluation.  Although  the  results  must  be  validated,  the  Quinn  et  al.  study 
provides  a  methodology  with  the  potential  to  permit  application  of  subjective 
procedures  during  the  earlier  stages  of  development  when  performance-based  and 
physiological  techniques  are  not  practicable. 


Taken  together,  implementation  requirements  can  therefore  impose  important 
constraints  on  the  use  of  the  various  classes  of  assessment  techniques  during  the 
development  process.  Instrumentation  and  facility  requirements  are  typically  more 
stringent  with  performance-based  and  physiological  techniques  than  with  subjective % 
procedures,  suggesting  the  use  of  the  latter  for  certain  situations.  V*VC 
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The  characteristic  of  operator  acceptance  is  important  to  ensure  that  an 
assessment  technique  will  yield  data  that  are  representative  of  the  load  Imposed 
by  the  task  or  system  function  in  question.  Assessment  procedures  which  are  per¬ 
ceived  by  operators  as  bothersome  or  artificial  incur  the  risk  of  being  ignored, 
performed  at  substandard  levels,  or  being  associated  with  significant  levels  of 
primary  task  intrusion.  Any  of  these  factors  can  lead  to  compromises  in  the 
effectiveness  of  a  technique. 

In  spite  of  the  potential  importance  of  operator  acceptance,  there  are  little 
or  no  formal  comparative  data  which  are  available  to  address  operator  reaction  to 
the  major  classes  of  techniques.  Although  some  Investigators  (e.g. ,  Hal Is ten  and 
Borg,  1975)  have  commented  on  operator  acceptance  of  a  number  of  procedures,  the 
data  are  not  sufficient  to  address  the  issue  in  a  comprehensive  manner.  Informal 

data  and  knowledge  of  the  procedures  involved  in  application  of  the  techniques 

can,  however,  be  used  to  provide  some  estimates  of  acceptance.  Informal  evidence, 
for  example,  suggests  that  subjective  procedures  usually  enjoy  a  high  degree  of 
user  acceptance,  quite  possibly  because  of  the  high  face  validity  associated  with 
many  current  rating  scales  (e.g..  Cooper  and  Harper,  1969;  Reid  et  al. ,  1981a). 

Operator  acceptance  should  also  be  quite  good  for  primary  task  measures,  since 

they  do  not  typically  involve  any  additional  operator  response  or  effort.  Physio¬ 
logical  techniques  would  have  some  potential  for  low  acceptance  if  the  recording 
instruments  used  are  considered  bothersome  by  the  operator,  but  this  does  not 
appear  to  have  been  a  significant  problem  with  most  techniques.  Secondary  task 
methods  could  also  be  considered  distracting  by  the  operator  if  the  requirement  to 
perform  the  secondary  task  interferes  with  primary  task  performance.  The  embedded 
secondary  task  technique  (Shingledecker  et  al.  ,  1980)  which  utilizes  a  secondary 
task  that  is  normally  performed  in  the  operational  environment  should,  however, 
minimize  this  risk. 


APPLICATION  GUIDELINES 

It  is  obvious  from  the  foregoing  discussion  that  no  single  assessment 

technique  is  capable  of  meeting  all  of  the  criteria  outlined  above.  The  various 
categories  of  techniques  are  characterized  by  the  capability  to  satisfy  some 
criteria,  but  not  others.  Criteria  vary  in  their  importance  as  a  function  of  the 
different  stages  of  design,  and  consequently,  techniques  vary  in  their  applicabil¬ 
ity.  It  is  therefore  clear  that  assessment  of  workload  across  the  various  phases 
of  the  design  process  will  require  the  complementary  use  of  multiple  metrics, 

since  no  single  metric  is  capable  of  providing  all  of  the  required  information. 

The  capability  of  individual  assessment  procedures  to  meet  the  various 

criteria  can  provide  some  guidance  regarding  their  use  for  specific  purposes  at 
different  stages  of  design.  Table  1  summarizes  the  current  status  of  the  proce¬ 
dures  with  respect  to  the  proposed  criteria,  and  can.be  used  as  a  basis  to  suggest 
particular  applications  for  each  class  of  technique. 

An  Investigator  requiring  a  nonintrusive  general  measure  of  load  in  an 
operational  environment  with  restricted  data  recording  capabilities  should,  for 
example,  consider  the  application  of  subjective  metrics.  On  the  other  hand, 
primary  task  measures  might  be  considered  for  application  in  a  high  fidelity 
simulator  with  performance  measurement  capability  when  the  objective  was  to 

evaluate  the  adequacy  of  operator  performance  with  a  particular  design  option. 
The  use  of  secondary  task  methodology  or  an  appropriate  physiological  technique  in 
a  system  simulator  would  be  u  iggested  if  the  Intent  was  to  Isolate  the  source  of 
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an  overload  that  had  been  previously  identified  through  use  of  a  subjective  or 
primary  task  metric.  The  potential  applications  for  each  class  of  metric  are 
clearly  much  more  extensive  than  those  suggested  by  these  hypothetical  situa¬ 
tions.  The  examples  do,  however,  illustrate  how  the  proposed  criteria  can  be 

applied  to  identify  the  class(es)  of  techniques  that  might  be  most  appropriate  for 
a  particular  application. 

In  many  instances,  use  of  the  proposed  criteria  will  result  in  simultaneous 
application  of  more  than  one  technique.  Applications  of  secondary  task  method¬ 
ology,  for  example,  require  the  measurement  of  primary  task  performance  in  order 
to  evaluate  the  degree  of  any  intrusion  that  might  have  occurred.  In  other 
instances,  the  objectives  of  an  evaluation  might  also  suggest  the  concurrent  use 
of  more  than  one  metric.  For  example,  a  comprehensive  evaluation  of  two  display 
options  might  include  the  use  of  both  primary  task  and  subjective  measures.  The 

primary  task  measure  would  permit  assessment  of  any  differences  in  the  adequacy  of 

task  performance  that  could  be  expected  with  the  options,  while  the  subjective 
technique  would  provide  Che  potential  to  identify  any  workload  differences  between 
the  options  that  were  not  reflected  in  the  less  sensitive  performance  measure. 

The  preceding  review  and  discussion  of  metrics  has  been  primarily  concerned 
with  classes  of  workload  assessment  techniques  in  general.  It  is  clear  from  the 
foregoing  discussion,  for  example,  Chat  the  general  category  of  subjective  metrics 
holds  a  great  deal  of  potential  for  use  during  system  design.  Once  a  class  of 
technique  has  been  identified  as  appropriate,  however,  an  individual  procedure  or 
measure  from  within  the  category  musC  be  chosen  for  actual  application.  Indi¬ 
vidual  procedures  themselves  can  also  vary  along  a  number  of  dimensions  that  can 
impact  their  suitability  for  use.  The  purpose  of  the  following  discussion  is  to 
briefly  review  some  recent  work  with  an  individual  subjective  assessment  technique 
that  appears  to  be  particularly  well  suited  for  a  number  of  applications  through¬ 
out  the  system  development  process. 


SUBJECTIVE  WORKLOAD  ASSESSMENT 


Subjective  workload  measurement  procedures  satisfy  a  number  of  the  criteria 
outlined  above  and,  as  a  consequence,  have  been  very  frequently  employed  as  work¬ 
load  assessment  techniques  (e.g. ,  Wllliges  and  Wierwille,  1979).  Despite  their 
advantages,  there  are  several  problems  which  have  been  traditionally  associated 
with  use  of  subjective  workload  metrics.  First,  in  many  applications,  individual 
rating  scales  have  been  developed  for  a  specific  investigation  and  have  not  been 
validated  for  generalized  use.  Second,  there  is  little  evidence  in  the  literature 
of  workload  rating  scales  that  have  been  rigorously  developed  on  the  basis  of 
psychometric  procedures  (e.g.,  Wllliges  and  Wierwille,  1979).  As  a  consequence, 
most  available  rating  scales  have  unknown  metric  properties,  and  must  be  assumed 
to  provide  only  ordinal  level  measurement. 


In  order  to  provide  a  workload  rating  scale  with  known  metric  properties  and 
with  the  potential  for  generalized  applicability,  a  procedure  termed  the  Subjec¬ 
tive  Workload  Assessment  Technique  (SWAT)  has  been  developed  (Reid  et  al.,  1981a; 
Reid,  Shingledecker,  Nygren,  and  Eggemeier,  1981b;  Reid,  Eggemeier,  and- 
Shlngledecker,  1982).  In  SWAT,  it  is  assumed  that  there  are  three  major  contribu¬ 
tors  to  subjective  mental  load:  (1)  time  load,  (2)  mental  effort  load,  and 

(3)  psychological  stress  load.  Time  load  refers  to  the  percentage  of  time  that  an 
operator  is  busy,  and  reflects  such  factors  as  overlap  and  interruption  among 
tasks.  Mental  effort  load,  on  the  other  hand,  refers  to  the  degree  of  attention 
or  concent r.-.t ion  required  during  task  performance.  The  final  dimension,  psycho¬ 
logical  St.-:3S  load,  reflects  any  additional  factors  that  cause  operator  anxiety 
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or  confusion  and,  therefore,  contribute  to  subjective  mental  load.  In  SWAT,  each 
of  the  three  dimensions  is  represented  by  an  individual  three-point  rating  scale 
with  verbal  descriptors  that  define  the  levels  on  each  dimension. 

A 

SWAT  is  based  on  application  of  conjoint  measurement  and  scaling  (e.g., 
Nygren,  1982).  Conjoint  measurement  and  scaling  permit  ratings  on  the  three 
dimensions  to  be  combined  into  one  overall  scale  of  workload  with  interval  meas¬ 
urement  properties.  In  order  to  identify  the  rule  which  is  appropriate  for  com¬ 
bining  the  three  dimensions  into  the  overall  interval  scale,  a  scale  development 
phase  is  completed.  During  this  phase,  subjects  rank-order  the  subjective  load 
associated  with  Che  27  possible  combinations  that  result  from  the  three  levels  of 
time,  mental  effort,  and  psychological  stress  load.  This  rank-ordering  informa¬ 
tion  is  subjected  to  a  series  of  axiom  tests  to  identify  the  rule  for  combining 
the  three  dimensions.  When  the  rule  has  been  established,  conjoint  scaling  is 
applied  to  derive  the  overall  scale  of  workload.  Subsequent  to  the  scale  develop¬ 
ment  phase,  subjects  participate  in  an  event  scoring  phase.  During  event  scoring, 
subjects  perform  the  task  or  mission  segment  of  interest  and  rate  the  time,  mental 
effort,  and  stress  load  associated  with  performance.  The  ratings  on  the  indi¬ 
vidual  dimensions  are  then  converted  to  one  of  the  27  points  on  the  Interval  scale 
that  was  derived  during  scale  development.  More  extensive  discussions  of  the 
scale  development  and  event  scoring  procedures  can  be  found  in  Reid  et  al. 

( 1981a, b) ,  and  Reid,  Eggemeler,  and  Nygren  (1982). 

One  aspect  of  the  work  conducted  during  the  development  of  SWAT  has  centered 
on  establishing  its  capability  to  reflect  workload  differences  in  a  number  of 
different  types  of  tasks  in  several  environments  that  are  representative  of  those 
found  during  system  development.  SWAT  has  been  successfully  applied  in  a  number 
of  laboratory  or  part-task  simulation  environments  (e.g.,  Reid  et  al.,  1981a; 
Eggemeler  et  al.  ,  1982,  1983;  Notestine,  1983);  in  several  full  mission  simulators  . 
(e.g.  ,  Reid,  Eggemeler,  and  Shingledecker,  1984;  Skelly,  Reid,  and  Wilson,  1983);  • 
and  under  conditions  that  are  similar  to  the  early  stages  of  system  development 
when  workload  estimates  must  be  based  on  detailed  mission  scenarios  and  descrip¬ 
tions  of  system  equipment  capabilities  (Quinn  et  al. ,  1982). 

Figure  1  illustrates  the  results  of  two  applications  of  SWAT  in  laboratory/ 
part-task  simulation  environments.  Panel  A  (Reid  et  al.  ,  1981a)  shows  the  results 
of  an  experiment  which  employed  several  levels  of  a  simulated  flight  control 
(critical  tracking,  Jex  and  Clement,  1979)  task  and  a  secondary  simulated  aircrew 
radio  comnwnications  task  (Shingledecker  et  al.,  1980).  Significant  differences 
in  SWAT  ratings  were  obtained  in  the  communication  task  alone  condition  versus  the 
more  difficult  dual  task  condition.  SWAT  ratings  also  successfully  discriminated 
levels  of  difficulty  in  both  the  simulated  flight  control  and  radio  communications 
tasks.  Panel  B  (Eggemeler  et  al.,  1983)  illustrates  the  effects  on  SWAT  ratings 
of  variations  in  the  rate  of  stimulus  presentation  in  a  sequential  short-term 
memory  task.  Subjects  in  the  experiment  were  required  to  monitor  a  visual  display 
and  update  the  status  of  four  categories  of  information  that  changed  at  several 
rates.  The  memory  task  was  intended  to  be  representative  of  the  demands  placed  on 
air  traffic  controllers  while  monitoring  flight  control  displays.  SWAT  ratings 
successfully  discriminated  levels  of  difficulty  in  the  memory  task,  even  though  a 
primary  task  measure  of  performance  errors  showed  no  significant  differences 
between  conditions. 

Several  recent  experiments  also  support  the  applicability  of  SWAT  to  full 
mission  simulation  environments.  SWAT  ratings  have  proven  sensitive  to  expected 
workload  variations  in  high  fidelity  flight  simulation  evaluations  of  advanced,-/ 
control/display  options  In  both  fighter  (Reid  et  al. ,  1984)  and  bomber  (Skelly  ^ 
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Figure  1.  Mean  SWAT  Racings  as  a  Function  of  Task  Difficulty  in  Two  Experiments. 

(Panel  A  illustrates  the  effects  of  simple  and  complex  radio  communica¬ 
tions  on  SWAT  ratings  in  both  single  and  dual  task  conditions  (Figure 
drawn  from  the  data  of  Reid  et  al.,  1981a).  Panel  B  shows  the  effect, 
of  stimulus  presentation  rate  manipulations  in  a  sequential  short-term 
memory  task  (Figure  adapted  from  Eggemeier  et  al.,  1983).] 
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et  al.  ,  1984)  aircraft.  Reid  et  al.,  for  example,  obtained  significant  differ¬ 
ences  in  pilot  SWAT  ratings  as  a  function  of  variations  in  the  number  of  opponents 
during,  a  fighter  mission.  SWAT  ratings  in  the  Skelly  et  al.  study  also  showed 
differences  that  were  logically  defensible  and  consistent  with  expectations. 
Pilot  ratings,  for  instance,  were  generally  higher  than  copilot  ratings,  except  vXJS 
for  a  number  of  segments  in  which  the  copilot  was  flying  the  aircraft.  Segments 
of  simulated  mission  which  included  various  types  of  threats  to  the  aircraft  were 
rated  higher  than  baseline  segments  that  did  not  include  such  threats*  In  both 
applications,  pilot  acceptance  of  the  rating  procedure  was  very  high,  and  in  both 
instances,  SWAT  ratings  were  taken  with  minimal  intrusion  by  having  the  pilot 
verbally  report  ratings  after  completion  of  a  mission  segment  to  an  experimenter 
stationed  outside  the  cockpit. 

The  Quinn  et  al.  (1982)  experiment  that  was  briefly  discussed  earlier  also 
utilized  the  SWAT  methodology  in  a  novel  application  of  the  technique.  The  pur¬ 
pose  of  the  Quinn  et  al.  study  was  to  evaluate  a  variety  of  methods  for  enhancing 
fighter  aircraft  systems,  Including  advanced  display,  control,  and  navigational 
concepts.  A  methodology  was  devised  to  comparatively  evaluate  the  enhancements 
along  a  number  of  dimensions  prior  to  prototype  development,  and  SWAT  was  Included 
to  quantify  predicted  effects  on  pilot  workload.  A  number  of  experienced  fighter 
pilots  were  provided  with  a  mission  scenario  and  detailed  descriptions  of  an 
advanced  baseline  version  of  the  aircraft  and  several  enhancements.  On  the  basis 
of  the  Information,  the  pilots  provided  mission  SWAT  ratings  for  the  various  ver¬ 
sions  of  the  baseline  system  that  Included  several  combinations  of  enhancements. 

The  interval  level  data  that  are  obtained  from  the  SWAT  procedure  permitted  use  of 
the  resulting  workload  ratings  in  a  multiattribute  utility  analysis  with  other 
factors  (cost,  system  performance)  to  permit  selection  of  several  options  for 
further  research.  Although  it  is  clear  that  the  results  of  the  projective  SWAT 
ratings  must  be  validated,  the  methodology  employed  by  Quinn  et  al.  is  significant  .. ■> 
in  that  It  demonstrates  the  feasibility  of  obtaining  SWAT  racings  on  the  basis  of  £  ^ 
detailed  mission  and  equipment  Information.  Use  of  the  technique  in  this  manner 
includes  obvious  time  and  cost  advantages  and,  as  noted  previously,  demonstrates 
the  potential  for  application  of  SWAT  during  the  earlier  stages  of  system  design. 

Taken  together,  the  results  of  current  work  with  the  SWAT  technique  clearly 
support  its  sensitivity  to  a  variety  of  tasks  that  are  relevant  to  system  opera¬ 
tion.  The  available  evidence  also  indicates  that  SWAT  has  a  very  high  potential 
for  applicability  across  several  stages  of  design.  These  data,  coupled  with  the 
advantages  of  the  interval  level  measurement  afforded  by  the  technique,  strongly 
support  the  utility  of  the  SWAT  metric  for  evaluation  of  workload  during  the  sys¬ 
tem  development  process. 


SUMMARY  AND  CONCLUSIONS 

Application  of  the  proposed  criteria  to  the  major  categories  of  workload 
assessment  techniques  indicates  that  a  battery  of  performance-based,  subjective, 
and  physiological  metrics  will  be  required  to  meet  the  varied  needs  for  workload 
aeasurement  that  arise  during  the  system  development  process.  In  many  Instances, 
the  capabilities  of  one  technique  supplement  those  of  another  procedure,  sug¬ 
gesting  the  complementary  use  of  the  various  metrics  at  different  stages  of 
design.  Among  the  classes  of  assessment  procedures  reviewed  above,  subjective 
techniques  appear  to  have  the  greatest  potential  for  application  across  the  vari¬ 
ous  phases  of  the  design  process,  and  the  SWAT  technique  is  one  such  procedure 
that  has  demonstrated  high  levels  of  sensitivity  and  applicability.  vS*' 
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Although  current  Information  is  sufficient  to  suggest  some  applications  for 
the  various  categories  of  techniques,  more  extensive  data  are  needed  to  refine 
proceuures  for  choice  of  a  metric  for  particular  applications.  For  example,  more 
complete  comparative  data  on  the  relative  sensitivity  and  intrusiveness  that  can 
be  expected  from  individual  techniques  from  within  particular  categories  (e.g., 
secondary  task)  of  procedures  represent  a  need  in  this  area.  As  was  noted 
previously,  several  such  efforts  have  been  recently  undertaken  (e.g., 
Shingledecker  et  al.,  1983;  Wierville  and  Casali,  1983b),  and  the  results  should 
provide  a  more  refined  basis  for  choice  of  metric  for  particular  applications.  An 
additional  area  requiring  further  experimentation  deals  with  the  extension  of  cur¬ 
rent  classes  of  metrics  to  the  earlier  and  later  stages  of  the  design  process. 
Implementation  requirements  have  somewhat  limited  the  applicability  of  secondary 
task  and  physiological  metrics  in  the  early  and  latter  stages  of  system  design, 
and  more  work  is  required  to  evaluate  the  application  of  these  techniques  beyond 
the  laboratory  and  simulation  environments.  Some  of  this  type  of  work  (e.g., 
Schifflet  et  al.,  1982)  has  been  conducted,  but  additional  efforts  are  required. 
Further  evaluation  and  extension  of  the  Quinn  et  al.  (1982)  procedure  for  applica¬ 
tion  of  subjective  metrics  during  the  early  stages  of  design  should  also  be 
pursued  in  order  to  supplement  available  analytic  and  modeling  procedures  that 
provide  the  current  capability  for  workload  assessment  during  this  phase  of  the 
development  process. 
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ABSTRACT 

Based  on  a  review  of  the  current  literature,  a  conceptual  framework  which 
incorporates  major  elements  related  to  operator  workload  has  been  devel¬ 
oped.  The  framework  treats  workload  as  a  multidimensional  construct  with 
important  physiological,  subjective,  and  behavioral  components.  An  impor¬ 
tant  implication  of  the  framework  is  that,  at  present,  a  comprehensive 
workload  assessment  methodology  should  include  a  number  of  measures, 
including  subjective,  physiological,  and  performance-based  metrics. 

INTRODUCTION 

A  primary  concern  of  human  factors  engineering  during  system  development 
and  evaluation  is  to  assure  that  the  performance  demands  imposed  by  a 
system  do  not  exceed  the  human  operator's  capacity  to  process  informa¬ 
tion.  Mental  workoad  is  the  term  which  has  been  used  in  referring  to  the 
degree  or  percentage  of  the  operator's  information  processing  capacity 
which  is  expended  in  meeting  system  demands.  The  growing  complexity  of 
modern  systems  and  associated  increases  in  workload  have  increased  the 
likelihood  of  exceeding  or  approaching  the  limitations  of  an  operator's 
processing  capacity.  As  a  consequence,  the  need  for  reliable  and  sensitive 


methods  of  assessing  the  workload  imposed  by  alternative  design  options  has 
assumed  increased  importance  in  recent  years. 


In  order  to  address  the  need  for  viable  workload  metrics,  a  large  number  of 
individual  measurement  techniques  have  been  developed  and  documented  in  a 
number  of  reviews  (e.g.,  Gartner  and  Murphy,  1976;  Willi ges  and  Wierwille, 
1979;  Moray,  1982).  Willi ges  and  Wierwille  (1979),  for  example,  identified 
28  specific  workload  assessment  techniques  which  have  been  proposed  in  the 
recent  literature.  Despite  the  fact  that  there  are  numerous  individual 
techniques,  most  workload  measures  can  generally  be  classified  as  belonging 
to  one  of  the  three  major  categories: 

1.  Subjective  measures,  such  as  rating  scales,  which  require  the 
operator  to  rate  or  somehow  characterize  the  subjective  workload 
associated  with  performance  of  a  particular  task  or  with  a  system 
design  option. 

2.  Psychophysiological  measures,  such  as  heart  rate  variability, 
which  derive  an  index  of  workload  from  some  aspect  of  the  opera¬ 
tor's  physiological  response  to  task  or  system  demands. 

3.  Performance-based  measures,  such  as  secondary  task  methodology, 
which  use  some  aspect  of  the  operator's  capability  to  perform 
within  the  system  as  a  measure  of  workload. 

All  three  classes  of  measures  have  been  used  extensively  during  the  past 
10  years  with  varying  degrees  of  success.  Given  the  large  number  of  mea¬ 
sures,  a  major  issue  in  developing  a  workload  assessment  methodology  is 
choosing  those  measures  that  should  be  incorporated  into  the  methodology. 
Unfortunately,  although  there  is  considerable  agreement  in  the  literature 
regarding  the  importance  of  workload,  there  has  not  been  substantial  agree¬ 
ment  concerning  the  most  appropriate  means  by  which  to  assess  it.  Much  of 
the  failure  to  agree  on  the  appropriate  assessment  technique  stems  from  the 
fact  that  workload  is  a  multidimensional  concept  (e.g.,  Johannsen,  Moray, 
Pew,  Rasmussen,  Sanders,  and  Wickens,  1979;  Sanders,  1979;  Willi ges  and 
Wierwille,  1979;  Sheridan  and  Simpson,  1979;  White,  1971)  which  has  been 
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used  in  referring  to  several  different  aspects  of  system  or  task  demand, 
operator  effort  and  information  processing  capacities,  operator 
performance,  and  systems  performance  (e.g.,  Sheridan  and  Stassen,  1979). 

In  order  to  provide  a  means  for  organizing  the  numerous  factors  associated 
with  workload,  a  general  conceptual  framework  which  addresses  major  ele¬ 
ments  of  the  workload  construct  has  been  developed.  The  major  objective  in 
assembling  the  framework  was  to  help  guide  the  development  of  a  comprehen¬ 
sive  workload  assessment  methodology  which  would  tap  essential  elements  of 
the  workload  construct.  The  framework  is  largely  based  on  current  con¬ 
ceptualizations  of  workload  and  theories  regarding  the  nature  of  capacity 
limitations  within  the  human  information  processing  system.  The  purpose  of 
this  paper  is  to  provide  an  overview  of  the  conceptual  framework  and  to 
discuss  implications  of  the  framework  for  workload  assessment. 

CONCEPTUAL  FRAMEWORK  FOR  WORKLOAD 

Despite  the  diversity  in  emphasis  among  various  approaches  to  workload, 
three  general  elements  which  represent  major  factors  in  most  current 
theoretical  treatments  can  be  identified.  These  general  elements  include: 


1.  Some  characterization  of  the  demands  placed  on  the  operator  by 
the  system  or  task. 

2.  Some  expression  of  the  capacity  or  effort  expenditure  required  by 
the  operator  to  deal  with  the  demands. 

3.  The  level  of  operator  performance  that  results  from  the  inter¬ 
action  of  task  demands  and  capacity/effort  expenditure. 

Figure  1  illustrates  these  major  elements  and  their  interrelationship  at  a 
very  general  level. 
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Insert  Figure  1  here 


As  indicated  in  Figure  1,  system  demands  are  imposed  on  the  operator  and 
must  be  related  to  or  mapped  onto  the  operator's  processing  capacity  or 
resources.  Depending  upon  the  efficiency  of  the  operator  in  mapping 
demands  onto  available  capacities  and  the  degree  of  effort  expended  by  the 
operator,  a  particular  level  of  performance  results.  Effort  is  intended  to 
represent  a  construct  similar  to  that  described  by  Kahneman  (1973)  and 
Jahns  (1973),  and  represents  a  nonspecific  input  which  is  required  to 
activate  information  processing  structures.  Most  conceptualizations  of 
workload  include  each  of  these  elements,  and  a  variety  of  workload  assess¬ 
ment  procedures  have  been  developed  to  evaluate  aspects  of  each  element. 

Several  current  models  and  theoretical  statements  concerning  workload 
(e.g.,  Jahns,  1973;  Wei  ford,  1978;  Sanders,  1979;  Sheridan  and  Simpson, 
1979;  Johannsen  et  al . ,  1979)  treat  major  elements  outlined  in  Figure  1  In 
more  detail.  These  models  and  statements  generally  maintain  that  workload 
is  clearly  a  multidimensional  construct  that  reflects  the  interaction  of 
such  elements  as  task  and  system  demands,  operator  processing  capacities 
and  effort,  subjective  performance  criteria,  operator  information  pro¬ 
cessing  strategies,  and  operator  training  or  experience. 

In  addition  to  suggesting  that  workload  itself  is  multidimensional,  a 
number  of  investigators  have  also  maintained  that  several  elements  of 
workload  depicted  in  Figure  1  are  themselves  multidimensional. 

Jahns  (1973),  for  instance,  categorized  sources  of  input  load  or  system 
demand  into  three  classes:  (1)  environmental,  (2)  situational,  and 
(3)  procedural.  Environmental  demands  included  factors  such  as  tempera¬ 
ture,  humidity,  noise,  and  acceleration  which  can  serve  as  sources  of  load 
for  an  operator.  Demands  that  were  character!- zed  as  situational  were 


composed  of  elements  such  as  display  and  control  characteristics,  display- 
control  arrangement,  vehicle  dynamics,  and  crewstation  volume.  These  types 
of  demand  reflect  traditional  human  factors  engineering  concerns,  and  play 
a  central  role  in  determining  the  overall  level  of  input  load  experienced 
by  an  operator.  Finally,  procedural  demands  included  elements  such  as 
mission  or  task  duration,  standard  system  operating  procedures,  the  mission 
itself,  and  briefings/instructions  given  to  the  operator.  Jahns  does  not 
specify  how  the  various  sources  of  demand  combine  to  determine  overall 
levels  of  load,  but  it  is  clear  that  input  load  is  cbnsidered  to  be 
multidimensional  and  is  described  as  a  vector  rather  than  a  scalar 
quanti  ty. 

A  similar  conclusion  has  been  reached  by  Johannsen  et  al .  (1979)  with 
respect  to  the  concept  of  effort  (Kahneman,  1973;  Jahns,  1973).  Johannsen 
et  al .  (1979)  pointed  out  that  the  notion  of  effort  is  extremely  complex 
and  can  be  interpreted  in  several  different  ways.  One  sense  of  effort 
identified  by  Johannsen  et  al .  is  related  to  the  physiological  activation 
as  measured  by  a  number  of  indices  ( e . g. ,  muscle  tension,  respiration  rate) 
that  occurs  when  an  operator  is  exposed  to  progressive  increases  in  mental 
load.  A  second  related  sense  of  effort  is  the  subjective  feeling  experi¬ 
enced  by  an  operator  under  a  high  load  condition.  Johannsen  et  al .  assume 
that  such  feelings  represent  the  products  of  muscular  tension  and  changes 
in  physiological  variables  such  as  blood  pressure  and  heart  rate.  It  is 
noted  that  an  operator  may  actually  feel  loaded  and  effortful  despite  the 
fact  that  there  is  no  change  in  the  adequacy  of  performance.  This  can  be 
explained  by  assuming  that  increases  in  task  difficulty  lead  to  a  period 
during  which  the  operator  is  working  harder  in  order  to  preclude  perform¬ 
ance  decrements.  The  clear  implication  is  that  effort  has  both  physio¬ 
logical  and  subjective  dimensions,  each  of  which  may  reflect  increases  in 
load  prior  to  any  actual  decrements  in  operator  performance. 

Operator  processing  capacity  which  is  depicted  in  Figure  1  has  also  been 
characterized  as  multidimensional  in  some  recent  descriptions  of  the  infor¬ 
mation  processing  system.  This  position  assumes  that  the  information 
processing  system  may  be  described  as  a  series  of  internal  processing 
structures,  each  with  its  own  processing  capacity  or  resources  which  are 


not  exchangeable  with  any  other  structure.  Current  theoretical  positions 
th^t  are  consistent  with  this  model  are  the  multiple  resources  or 
structure-speci fic  resource  models  of  processing  capacity  (e.g.,  Wickens, 
1979,  1981;  Navon  and  Gopher,  1979;  Sanders,  1979;  North,  1977;  Kantowitz 
and  Knight,  1976).  According  to  this  theory,  a  considerable  amount  of 
capacity  may  remain  unused  in  responding  to  a  particular  task's  demands, 
because  ony  a  limited  number  of  processing  structures  may  be  involved  in 
that  response.  An  important  consequence  of  this  position  identified  by 
Sanders  (1979),  Wickens  (1979),  and  Gopher  (1978),  is  that  mental  load 
cannot  be  conceptualized  as  a  single  dimension;  and  ultimately,  a  task  may 
have  to  be  described  in  terms  of  multidimensional  patterns  of  mental 
load.  A  central  question  in  workload  specification  becomes  one  of 
determining  the  extent  to  which  various  processing  resources  are  involved 
in  a  task  which,  in  turn,  specifies  the  pattern  of  mental  load.  Several 
theorists  (e.g.,  North,  1977;  Sanders,  1979)  have  suggested  candidate 
dimensions  of  resources  that  could  be  used  in  describing  the  resource 
demand  composition  of  a  task.  The  most  comprehensive  position,  however,  is 
that  of  Wickens  (1931)  who,  on  the  basis  of  available  evidence,  has 
identified  three  primary  dichotomous  dimensions  that  appear  to  define 
separate  resources.  These  dimensions  include: 

1.  Stages  of  information  processing  (perceptual /central  processing 
operations  versus  response  selection  and  execution). 

2.  Modalities  of  perception  (auditory  versus  visual). 

3.  Codes  of  information  processing  and  response  (spatial -manual 
versus  verbal -vocal ) . 

Under  the  system  proposed  by  Wickens,  an  adequate  description  of  the  load 
imposed  by  a  task  should  include  sped  fication  of  .its  demand  composition  on 
each  of  the  three  dimensions.  The  implication  of  the  multiple  resources 
theory  is  clear:  the  capacity  to  process  information  is  multidimensional 
and  characterizations  of  the  load  imposed  by  a  task  should  reflect  that 
multidimensionali ty. 
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The  general  conceptual  framework  outlined  previously  can  now  be  modified  to 
be  more  descriptive  of  elements  related  to  workload.  Figure  2  illustrates 
the  modified  conceptual  framework.  In  the  modified  framework,  workload  and 
operator  performance  continue  to  represent  the  product  of  an  interaction  of 
several  factors.  In  accordance  with  Jahns'  position,  system  demands  are 
represented  as  multidimensional,  Including  the  classes  of  environmental, 
situational,  and  procedural  load.  Effort  is  also  conceptualized  as 
reflecting  several  dimensions,  including  strong  physiological  and  subjec¬ 
tive  components.  Likewise,  operator  processing  resources  have  been  revised 
to  reflect  several  of  the  dimensions  represented  in  the  multiple  resources 
view  of  the  information  processing  system. 


Insert  Figure  2  here 

The  major  conclusion  that  follows  from  the  present  framework  is  that  work¬ 
load,  as  currently  conceptualized,  represents  a  multidimensional  construct 
that  includes  important  physiological,  subjective,  and  behavioral  com¬ 
ponents.  The  multidimensionality  is  reflected  not  only  In  the  interaction 
of  several  elements  to  determine  levels  of  load,  but  also  in  the  multi¬ 
dimensional  i  ty  of  several  elements  themselves.  The  multidimensional  nature 
of  the  framework  has  important  implications  for  workload  assessment,  and 
these  are  discussed  in  the  next  section. 

IMPLICATIONS  OF  THE  CONCEPTUAL  FRAMEWORK  FOR  WORKLOAD  ASSESSMENT 

A  major  implication  of  the  current  framework  is  that  no  single  measurement 
technique  will  provide  a  comprehensive  means  for  assessment  of  load.  Since 
current  theory  generally  maintains  that  there  are  important  physiological, 
subjective,  and  behavioral  components  of  load,  a  comprehensive  approach  to 
assessment  of  workload  should  include  physiological,  subjective,  and 
behavioral  or  performance-based  measures.  At  present,  it  is  not  clear  how 
such  measures  might  ultimately  be  combined  to  provide  a  multidimensional 
index  of  load,  but  it  should  be  clear  from  the  foregoing  discussion  that 
subjective  and  physiological  measures  can  potentially  provide  information 
not  afforded  by  performance-based  measures,  and  vice  versa.  Therefore,  it 
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appears  that  the  most  viable  approach  to  comprehensive  workload  assessment 
would  be  a  battery  of  measures,  including  subjective,  physiological ,  and 
performance-based  components  which  could  be  applied  to  derive  indices  of 
several  components  of  load.  Similar  conclusions  have  been  drawn  recently 
by  several  others.  Johannsen  et  al .  (1979),  for  example,  concluded  that 
workload  contains  behavioral,  performance,  physiological,  and  subjective 
components  and  indicated  that  appropriate  measures  would  be  required  for 
each  component.  Willi ges  and  Wierwille  (1979),  in  their  review  of 
behavioral  workload  assessment  procedures,  maintain  that  due  to  the  multi¬ 
dimensionality  of  workload,  it  appears  unlikely  that  any  single  measure 
will  be  completely  sufficient  for  characterizing  load.  Willi ges  and 
Wierwille  conclude  that  multiple  measures,  includng  dimensions  of  subjec¬ 
tive  opinion,  spare  mental  capacity,  primary  tasks,  and  physiological 
correlates  need  to  be  considered.  Similarly,  Jahns  (1973)  noted  that 
because  of  the  complex  interactions  involved  in  determining  levels  of 
operator  effort,  a  broad  spectrum  of  measurement  techniques  for  operator 
workload  need  to  be  investigated.  White  (1971)  also  indicated  that  work¬ 
load  is  multidimensional  and,  at  present,  cannot  be  defined  adequately  in 
terms  of  any  single  measure. 

Further  support  for  the  necessity  of  considering  multiple  measures  of  load 
comes  from  the  fact  that  when  several  measures  of  workload  are  applied  in  a 
situation,  they  commonly  exhibit  some  degree  of  dissociation  (e.g.,  Borg, 
1978;  Hicks  and  Wierwille,  1979;  Dornic  and  Andersson,  1980;  Wickens  and 
Derrick,  1981;  and  Moray,  1982).  This  type  of  result  can,  of  course,  be 
interpreted  within  the  multidimensional  workload  framework  outlined  above, 
since  different  measurement  techniques  can  be  assumed  to  be  maximally 
sensitive  to  different  dimensions  of  load.  Wickens  and  Derrick  (1981),  for 
instance,  have  noted  that  a  lack  of  correspondence  among  workload  measures 
can  be  attributed  to  the  multidimensional  nature  of  information  processing 
resources  that  are  assumed  to  underlie  performance  and  workload.  Based  on 
the  results  of  several  experiments  in  which  dissociation  occurred  between 
physiological,  primary  task,  subjective,  and  secondary  task  measu  es, 
Wickens  and  Derrick  proposed  that  some  measures  may  be  considered  more 
generally  sensitive  to  overall  levels  of  demand  for  resources  anywhere 
within  the  Information  processing  system,  while  others  are  more  diagnostic 
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in  the  sense  of  specifically  reflecting  demands  imposed  on  particular 
resources  (e.g.,  percpetual  versus  motor  output)  within  the  processing 
system.  It  was  suggested  that  some  physiological  measures  such  as  heart 
rate  variability  might  be  sensitive  to  the  total  demand  placed  on  all 
resources  of  the  system,  even  if  the  demand  was  imposed  such  that  no  single 
resource  was  overloaded  and  no  performance  decrement  occurred.  Subjectve 
measures  were  also  viewed  as  generally  sensitive  to  demands  imposed  any¬ 
where  in  the  system,  while  secondary  task  measures  were  thought  to  be  more 
diagnostic  in  providing  information  regarding  the  specific  resources 
demanded  by  a  task. 

Wickens  and  Derrick  concluded  that  since  individual  measures  might  provide 
different  types  of  information  about  load,  choice  of  measure  might  be 
dictated  in  part  by  the  purpose  of  taking  the  measure.  If  the  total  load 
imposed  by  a  task  or  system  were  to  be  assessed,  some  physiological  or 
subjective  techniques  might  be  used  in  conjunction  with  primary  task  per¬ 
formance  measures.  However,  if  the  specific  locus  of  an  overload  (e.g., 
central  processing  versus  response  execution)  were  to  be  identified  in 
order  to  provide  more  diagnostic  information  to  a  human  factors  engineer, 
then  secondary  task  methodology  might  be  the  more  appropriate  choice.  Note 
that  in  this  type  of  scheme,  several  measurement  techniques  could  be  used 
in  a  complementary  fashion,  with  some  subjective  or  physiological  measures 
providing  an  initial  index  of  overall  task  or  system  workload,  and  secon¬ 
dary  tasks  or  other  physiological  measures  (e.g.,  evoked  cortical 
potential)  being  used  subsequently  to  develop  more  precise  information 
regarding  the  locus  of  specific  overloads. 

The  clear  implication  that  follows  from  the  conceptual  framework  and  from 
the  noted  dissociation  of  workload  measures  is  that,  at  present,  a  compre¬ 
hensive  workload  assessment  methodology  will  require  physiological,  subjec¬ 
tive,  and  performance-based  measurement  techniques.  The  major  goals  for 
research  in  the  workload  metric  development  area  suggested  by  the  framework 
are  to:  [1]  initially  identify  the  most  sensitive  measure(s)  within  each 
category  of  assessment  technique;  and  [2]  conduct  a  systematic  comparison 
of  the  information  provided  by  each  category  of  assessment  technique  so 
that  optimal  combination(s)  of  measures  required  for  a  comprehensive  work¬ 
load  assessment  methodology  can  ultimately  be  established. 
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Subjective  Workload  Assessment  Technique  (SWAT) 

by  G.  Reid 

S'JAT  CARD  SORT  INSTRUCTIONS  FOR  SUBJECTS 

During  the  course  of  this  experiment,  you  will  be  asked  to  quantify  the  mental 
workload  required  to  complete  the  tasks  you  will  be  performing.  Mental 
Workload  refers  to  how  hard  you  work  to  accomplish  some  task,  groups  of  tasks, 
or  an  entire  job.  The  workload  imposed  on  ycu  at  any  one  time  consists  of  a 
combination  of  various  dimensions  which  contribute  to  the  subjective  feeling 
of  workload.  The  Subjective  Workload  Assessment  Technique  (SWAT)  defines 
these  "dimensions  as  1)  Time  Load,  2)  Mental  Effort  Load,  and  3)  Psychological 
Stress  Lead. 

For  the  purpose  of  SWAT,  the  three  dimensions  have  been  assigned  three  levels. 
The  dimensions  and  their  levels  are  described  in  the  following  paragraphs. 

TIME  LOAD 

Time  load  refers  to  the  fraction  of  the  total  time  that  ycu  are  busy.  When 
time  load  is  low,  sufficient  time  is  available  to  complete  all  of  your  mental 
work  with  some  time  to  spare.  As  time  load  increases,  spare  time  drops  out 
and  some  aspects  of  performance  overlap  and  interrupt  one  another.  This 
overlap  and  interruption  can  come  from  performing  more  than  one  task  or  from 
different  aspects  of  performing  the  same  task.  At  higher  levels  of  time  lead, 
several  aspects  of  performance  often  occur  simultaneously,  you  are  busy,  and 
interruptions  ape  very  frequent. 

Time  load  may  be  rated  on  the  three  point  scale  below. 

(1)  Often  have  spare  time.  Interruptions  or  overlap  among  activities  occur 
infrequently  or  not  at  all. 

(2)  Occasionally  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  frequently. 

(3)  Almost  never  have  spare  time.  Interruptions  or  overlap  among 
activities  are  very  frequent,  or  occur  all  the  time. 


MENTAL  EFFORT  LOAD 

As  described  above  time  load  refers  to  the  amount  of  time  one  has  available 
to  perform  a  task  or  tasks.  In  contrast,  mental  effort  load  is  an  index  of 
the  amount  of  attention  or  mental  effort  required  by  a  task  regardless  of 
the  number  of  tasks  to  be  performed  or  any  time  limitations.  When  mental 
effort  load  is  low,  the  concentration  and  attention  required  by  a  task  is 
minimal  and  performance  is  nearly  automatic.  As  the  demand  for  mental 
effort  increases,  due  to  task  complexity  or  the  amount  of  information  which 
must  be  dealt  with  in  order  to  perform  adequately,  the  degree  of 
concentration  and  attention  required  increases.  High  mental  effort  load 
demands  total  attention  or  concentration  due  to  task  complexity  or  the 
amount  of  information  that  must  be  processed. 

Mental  effort  load  may  be  rated  using  the  three  point  scale  below. 

(1)  Very  little  conscious  mental  effort  or  concentration  required. 
Activity  is  almost  automatic,  requiring  little  or  no  attention. 


1 


(2)  Moderate  conscious  mental  effort  or  concentration  required  Complexity 
of  activity  is  moderately  high  due  to  uncertainty,  unpredictability,  or 
unfamiliarity.  Considerable  attention  required. 

(3)  Extensive  mental  effort  and  concentration  are  necessary.  Very  complex 
activity  requiring  total  attention. 

PSYCHOLOGICAL  SSKESS  LOAD 


Stress  load  refers  to  the  contribution  to  total  workload  of  any  conditions 
that  produce  anxiety,  frustration  or  confusion  while  performing  a  task  or 
tasks.  At  low  levels  of  stress,  one  feels  relatively  relaxed.  As  stress 
increases,  confusion,  anxiety  or  frustration  increase  and  greater 
concentration  and  determination  are  required  to  maintain  control  of  the 
situation. 

Psychological  stress  load  may  be  rated  on  the  three  point  scale  below. 

(1)  Little  confusion,  risk,  frustration,  or  anxiety  exists  and  can  be 
easily  accommodated. 

(2)  Moderate  stress  due  to  confusion,  frustration,  or  anxiety  noticeably 
adds  to  workload.  Significant  compensation  is  required  to  maintain 
adequate  performance. 

(3)  High  to  very  intense  stress  due  to  confusion, .  frustration,  or  anxiety. 
High  to  extreme  determination  and  self-control  required. 

Each  of  the  three  dimensons  just  described  contribute  to  workload  during 
performance  of  a  task  or  group  of  tasks.  Note  that  although  all  three 
factors  may  be  correlated,  they  need  not  be.  For  example,  one  can  have  many 
tasks  to  perform  in  the  time  available  (high  time  load)  but  the  tasks  may 
require  little  concentration  (low  mental  effort.)  Likewise,  one  can  be 
anxious  and  frustrated  (high  stress)  and  have  plenty  of  spare  time  between 
relatively  simple  tasks.  Since  the  three  dimensons  contributing  to  workload 
are  not  necessarily  correlated,  please  treat  each  dimension  individually  and 
give  independent  assessments  of  the  time  load,  mental  effort  load,  and 
stress  load  that  you  experience  in  performing  the  following  tasks. 

One  of  the  most  important  features  of  SWAT  is  its  unique  scoring  system. 
SWAT  uses  a  procedure  to  find  separate  scoring  weights  for  each  level  of  a 
dimensions.  Then,  it  determines  a  distinctive  workload  scale  for  each 
person.  This  scaling  system  greatly  improves  the  precision  of  the  workload 
ratings  you  will  give  later. 

In  order  to  develop  your  individual  scale,  we  need  information  from  you 
regarding  the  amount  of  workload  you  feel  is  imposed  by  various  combinations 
of  the  dimensions  described  above.  We  get  this  information  by  having  you  rank 
order  the  workload  associated  with  each  of  the  combinations. 

In  order  for  you  to  rank  order  the  workload  for  each  of  the  combinations, 
you  have  been  given  a  set  of  27  cards  with  the  combinations  from  each  of  the 
three  dimensions.  Each  card  contains  a  different  combination  of  levels  of 
Time  Load,  Mental  Effort,  and  Psychological  Stress.  Your  job  is  to  sort  the 
cards  so  that  they  are  rank  ordered  according  to  the  level  of  workload 
represented  on  each. 


In  completing  your  card  sorts,  please  consider  the  workload  imposed  on  a 
person  by  the  combination  represented  in  each  card.  Arrange  the  cards  from 
the  lowest  workload  condition  through  the  highest  condition.  You  may  use 
any  strategy  you  choose  in  rank  ordering  the  cards.  One  strategy  that 
proves  useful  is  to  arrange  the  cards  into  a  number  of  preliminary  stacks 
representing  "High",  "Moderate",  and  "Low"  workload.  Individual  cards  can 
be  exchanged  between  stacks,  if  necessary,  and  then  rank  ordered  within 
stacks.  Stacks  can  then  be  recombined  and  checked  to  be  sure  that  they 
represent  your  ranking  of  lowest  to  highest  workload.  However,  the  choice 
of  strategy  is  up  to  you  and  you  should  choose  the  one  that  works  best  for 
you. 

There  is  no  "school  solution"  to  this  problem.  There  is  no  correct  order. 
The  correct  order  is  what,  in  your  judgment  best  describes  the  progression 
of  workload  from  lowest  to  highest  for  a  general  case  rather  than  any 
specific  event.  That  judgment  differs  for  each  of  us.  The  letters  you  see 
on  the  back  of  the  cards  are  to  allow  us  to  arrange  the  cards  in  a 
previously  randomized  sequence  so  that  everyone  gets  the  same  order.  If  you 
examine  your  deck  you  will  see  the  order  on  the  back  runs  from  A  through  Z 
and  then  ZZ. 

Please  remember: 


1) 


The  card  sort  is  being  done  so  a  workload  scale  may  be  developed  for  you. 
This  scale  will  have  a  distinct  workload  value  for  each  possible 
combinations  of  Time  Lead,  Mental  Effort  Load  and  Psychological  Stress 
Load. 


Time 


Effort 


Stress 


2) 


Workload  Scale 
0 


100 


When  performing  the  card  sorts,  use  the  descriptors  printed  on  the 
cards.  Please  remember  not  to  sort  the  cards  based  on  a  particular  task 
(such  as  flying  an  airplane).  Sort  the  cards  according  to  your  general 
view  of  workload  and  hew  important  you  consider  the  dimensions  of  time, 
mental  effort,  and  psychological  stress  lead  to  be. 


During  the  actual  experiment,  you  will  accomplish  the  desired  task.  Then, 
you  will  provide  a  SWAT  score  based  on  your  opinion  of  the  mental  workload 
required  to  perform  the  task.  This  SWAT  score  will  consist  of  one  number 
from  each  of  the  three  dimensions. 


v.v; 


For  example,  a  possible  SWAT  score  is  1  -  2  -  2.  This  represents  a  1  for 
Time  Load,  a  2  for  Mental  Effort  Load,  and  a  2  for  Psychological  Stress 
Load. 

4)  We  are  not  asking  for  your  preference  concerning  Time,  Mental  Effort  and 
Psychological  Stress  Load.  Some  people  may  prefer  to  be  "busy"  rather 
than  "idle"  in  either  the  Time  Load,  Mental  Effort  load,  or  Psychological 
Stress  Load  dimension.  We  are  not  concerned  with  this  preference.  We 
need  information  on  how  the  three  dimensions  and  the  three  levels  of  each 
one  will  affect  the  level  of  workload  as  you  see  it.  You  may  prefer  a 
2-2-2  situation  instead  of  a  1  -  1  -  1  situation.  But,  you  should 
still  realize  that  the  1-1-1  situation  imposes  less  workload  on  you 
and  leaves  a  greater  reserve  capacity. 

From  this  point  until  you  have  completed  the  sorting  will  probably  take 
30  minutes  to  an  hour. 

Please  feel  free  to  ask  questions  at  any  time.  Thank  ycu  for  your 
cooperation. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Moderate  conscious  mental  effort  or  concentration 
required.  Complexity  of  activity  is  moderately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Moderate  conscious  mental  effort  or  concentration 
required.  Complexity  of  activity  is  moderately  high  due 
to  uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 


High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Moderate  conscious  mental  effort  or  concentration  required 
Complexity  of  activity  is  m^d?rately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Moderate  conscious  mental  effort  or  concentration  required 
Complexity  of  activity  is  moderately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 


High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 
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Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Moderate  conscious  mental  effort  or  concentration  required, 
Complexity  of  activity  is  moderately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 


Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 
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Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Moderate  conscious  mental  effort  or  concentration  required 
Complexity  of  activity  is  moderately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required 

to  maintain  adequate  performance. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 


High  to  very 
or  anxiety, 
self-control 


intense  stress  due  to  confusion,  frustration, 
High  to  extreme  determination  and 
required . 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 


Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Moderate  conscious  mental  effort  or  concentration  required 
Complexity  of  activity  is  moderately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 


High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Moderate  conscious  mental  effort  or  concentration  required 
Complexity  of  activity  is  moderately  high  due  to 
uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 

Moderate  stress  due  .to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 


Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 


Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 
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Almost  never  have  spare  time.  Interruptions  or  overlap 
among  activities  are  very  frequent,  or  occur  all  the  time. 

Very  little  conscious  mental  effort  or  concentration 
required.  Activity  is  almost  automatic,  requiring  little 
or  no  attention. 

High  to  very  intense  stress  due  to  confusion,  frustration, 
or  anxiety.  High  to  extreme  determination  and 
self-control  required. 


Occasionally  have  spare  time.  Interruptions  or  overlap 
among  activites  occur  frequently. 

Extensive  mental  effort  and  concentration  are  necessary. 
Very  complex  activity  requiring  total  attention. 

Moderate  stress  due  to  confusion,  frustration,  or  anxiety 
noticably  adds  to  workload.  Significant  compensation  is 
required  to  maintain  adequate  performance. 


Often  have  spare  time.  Interruptions  or  overlap  among 
activities  occur  infrequently  or  not  at  all. 

Moderate  conscious  mental  effort  or  concentration 
required.  Complexity  of  activity  is  moderately  high  due 
to  uncertainty,  unpredictability,  or  unfamiliarity. 
Considerable  attention  required. 

Little  confusion,  risk,  frustration,  or  anxiety  exists  and 
can  be  easily  accommodated. 


MAIN  MENU 

As  can  be  seer,  in  Figure  1,  this  screen  has  three  main 
functions.  The  first  is  to  allow  entering  of  date,  study 
name,  and  additional  comments  for  labeling  a  data  set.  The 
second  function  is  to  specify  the  r. a n e  of  the  file  to  be  use 
tc  later  identify  the  data  set  and  the  third  function  is  to 
select  the  program  options. 

When  you  assign  the  file  name  c r.  this  screen  it  becomes 
the  file  the  program  uses  uhen  the  data  set  is  stored  to 
disk.  The  file  name  should  be  different  from  the  study  name 
or  some  variant  cf  the  study  name  tc  allow  for  multiple 
analysis  you  may  do  within  a  study.  As  always,  if  you  want 
to  retrieve  a  data  file  for  something,  the  file  name  you  use 
must  be  exactly  correct.  For  example,  you  may  war,:  tc  acces. 
a  previous  data  file  to  m o d i f y  it  by  a d d i r. g  additional 
subjects . 

to 

Upon  entering  a  new  file  name,  t  h  e  data  entry  scree:':, 
which  is  descritei  next,  will  appear.  If  ye-,  select  a  r.  a  m  e 
of  an  existing  file: 

a .  You  will  be  asked  to  verify  that  -you  wish  to  use 
data  from  a  previous  file.  This  is  a  safeguard  t  prevent 
accident ! y  writing  ever  an  existing  oats  file. 

b  .  The  screen  will  change  *  =  h  -  t  h  e  .to;-*,  e d  v  e  r  s  :  : 


.  r.  i  s  :s  a  same  guar 


of  the  Main  Menu  screen  including  the  information  .just- 
entered  by  you  as  depicted  in  Figure  2.  You  will  be  asked  t 
choose  one  of  the  following  options  to  direct  the  program  to 
specific  options  described  below: 

OPTIONS 

1.  Edit  comments-  This  option  allows  you  to 
change  any  or  all  of  the  comments  on  the  first  screen. 

2.  Data  entry-  This  option  takes  you  to  the 
data  entry  screen  to  add,  subtract,  or  otherwise  modify  data 

3.  Program  setup-  Selection  of  this  option 
takes  you  to  the  program  setup  screen  where  you  will  select 
options  that  permit  you  to  specify  the  analysis  you  want 
performed.  See  the  description  of  Program  setup  screen  for 
more  details. 

DATA  ENTRY  SCREEN: 


Upon  selection  of  "Data  Entry”  on  the  main  menu  screen  o 
when  you  enter  a  new  file  name,  a  formatted  screen  as 
depicted  in  Figure  3  will  be  displayed  that  allows  you  tc 
enter  cr  modify  data.  The  number  of  subjects  that  was 
previously  entered  on  the  main  menu  will  be  displayed  at  the 
top  of  the  screen,  and  sufficient  space  for  data  for  this 
number  of  subjects  will  be  created.  The  following  options 


Enter  Data 


If  you  select  this  option  the  cursor 


1 . 

automatically  moves  to  the  position  for  subject  #1,  card  1-1- 
1  and  data  may  be  entered.  The  rank  assigned  to  card  1-1-1 
by  this  subject  should  be  entered  and  then  the  cursor  will 
move  to  subject  #2.  The  program  is  set  up  tc  accept  the  rank 
assigned  by  all  subjects  fcr  card  1-1-1  before  the  cursor 
automatically  goes  tc  the  position  for  card  1-1-2  fcr  subject 
#1 .  In  this  manner  the  program  will  step  through  all  2? 
cards  for  the  specified  number  of  subjects.  You  may  manually 
override  this  sequence  and  move  the  cursor  to  any  position  on 
the  data  entry  screen  in  the  manner  that  any  screen  editor  is 
used.  Every  third  row  on  this  screen  is  highlighted  to  make 
it  easier  to  keep  track  of  where  you  are  as  ycu  scroll 
through  the  data.  As  you  reach  the  end  of  the  displayed 
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2.  Save  Data  -  When  you  select  this  option 
the  data  currently  cn  the  data  entry  screen 
hard  disk  file  with  the  name  y  c u  p r e v i o  u  s  1  y 
main  menu  screen.  When  the  data  are  saved, 
changes  to  that  of  Figure  4  and  the  program 
following  options: 


the  program  saves 
tc  a  diskette  or 
specified  cn  t h e 
this  screen 
displays  the 


a .  Print  data 
of  the  input  data  set 


This  allows  vcu  t  c  obtain  a  hard  copy 
for  easy  reference.  Since  the  entire 
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data  table  cannot  fit  c n  the  screen  at  one  time,  this  may  be 
a  more  desirable  way  to  proof  and  edit  your  input  data. 

b.  Edit  data  -  This  option  allows  you  to  position  the 
cursor  at  any  point  on  the  data  entry  screen  and  make  changes 
to  the  data. 

c.  Program  setup  -  This  option  directs  the  program  to 
proceed  to  the  next  screen  in  order  to  continue  the  normal 
analysis  process. 

3.  Edit  data  -  If  you  are  working  with  a  data  set  which  was 
previously  entered,  this  function  allows  you  to  return  to  the 
data  entry  table  and  make  changes  to  the  data.  Use  the  arrow 
keys  to  move  the  cursor  to  the  appropriate  position  so  that 


y  o  u  its  a  y  m  a 

ke  t! 

,e  desired  changes. 

Press! 

ng  < 

Return >  or  using 

the  arrow 

keys 

to  move  the  curser 

to  the 

next 

position  will 

enter  the 

data 

into  the  computer’ 

s  memory 

4.  Escape  -  This  option  takes  you  back  to  the  Main  Menu  and 
easily  allows  you  to  leave  the  program,  help  you  get 
reoriented  if  you  were  lost  or  confused,  or  make  changes  in 
earlier  screens.  When  this  option  is  chosen,  there  will  be  a 
prompt  to  ask  whether  you  want  to  save  any  data  entered  cn 
the  screen,  to  a  diskette  file. 

RE  REUSE?,  data  is  saved  0  A'  L  Y  through  the  SAVE  DATA  {  u  rc  t  :  on  , 


or  saving  during  an  ESCAPE. 
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PROGRAM  SETUP  SCREEN: 


This  screen  is  presented  in  Figure  5  and  gives  you 
options  for  the  types  of  analyses  to  be  performed.  Enter  the 
number(s)  of  the  analyses  which  you  want  performed.  There 
are  three  main  analyses: 

1.  Prototype  correlations  and  Kendall’s  Coefficient  of 
Concordance 

2.  Axiom  Testing 

3.  Scaling  Solution 

1.  The  Prototype  Correlations  analysis  performs  a  Spearman’s 
rank  order  correlation  on  each  of  the  subject’s  rank  order 
data  with  the  six  possible  ’’perfect”  rankings.  The  pattern 
of  correlations  indicates  the  importance  a  subject  places  on 
the  three  dimensions  of  the  SWAT  definition  of  workload.  The 
Kendall’s  Coefficient  of  Concordance  is  an  index  of  the 
degree  of  agreement  among  the  group  of  subjects  about  the 
order  of  the  27  cards.  A  high  Kendall’s  indicates 
substantial  agreement  about  the  order  and  therefore  about  the 
relative  importance  of  the  three  dimensions. 


2.  The  Axiom  testing  section  performs  the  axiom  tests  for 
independence,  joint  independence,  and  double  cancellation. 
This  is  done  to  check  for  violations  of  these  axioms  which 
may  invalidate  the  additive  model  as  being  a  suitable  me  del 
to  use  for  the  conjoint  scaling  routine.  Cnly  a  summary  of 
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the  three  axiom  test  is  automatically  disp.ayed,  but  the 
complete  history  of  results  for  all  the  tests  may  be  viewed 
or  printed  by  selection  of  the  appropriate  option. 

3.  Scaling  solution  -  Produces  the  scale  values  that  result 
from  the  conjoint  scaling  routine.  These  values  are  then 
used  as  the  work  Scad  scores  in  subsequent  analyses  if  results 
£ ros  the  Event  Scoring  Phase. 

There  are  three  met  h  c  d  s  £  c  r  h  a  r.  d  ling  t  h  e  scale  development 
c a  t  a ;  group  solution,  prototyped  solutions,  and  individual 
solutions.  The  determination  of  which  solution  is  best  is 
based  on  study  objectives  and  the  results  of  the 
correlations^  Kendall's  Coefficient,  and  axiom  tests.  For  a 
mere  detailed  explanation  of  interpreting  these  analyses, 
refer  to  the  section  entitled  "Interpretation  c £  Output".  If 
prototype  or  individual  axiom  tests  or  scaling  solutions  are 
chosen,  the  program  will  ask  for  specific  prototype  groups 
and/or  individual  subjects  you  desire  to  include  in  the 
analysis.  This  option  is  useful  if,  for  example,  you  wish  t c 
run  three  subjects  individually.  Their  subject  number  may  be 
specified  arc  the  program  will  only  include  their  analysis. 
If.  for  example,  you  specify  a  prototype  solution  by 
separating  the  subjects  into  prototype  groups,  the  pregram 
will  use  the  suggested  prototype  groups  provided  in  the  next 
screen  to  be  described.  [If  you  wish  to  change  the  prototype, 
press  'Return-  t  c  procede  with  the  analysis.  1 
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evaluation  of  the  pattern  cf  correlations.  To  leave  a 
particular  subject  out  of  the  analysis,  simply  change 
subject’s  suggested  prototype  to  an  "L”.  The  program 
ignore  rhis  subject's  data.  This  option  can  be  used  i 
case  where  a  subject  belongs  to  a  particular  prcrot: 
tut  his/her  data  also  contains  a  large  number  cf  ax: 
violations,  indicating  an  unacceptable  amount  cf  e r : 
data.  In  this  situation  it  may  be  desirable  tc  exciuce  cm 
subject  from  the  scaling  solution,  or  attempt  to  remove  the 
ambiguity  in  the  data  through  additional  informatics 
from  the  subject.  T c  end  any  changes  to  the  protot: 
simply  push  FI  tc  return  to  the  rest  cf  the  options 


3 .  Return  t  c  Program  Setup  - 


noe  program  s  e  t  u ; 


This  option  allows  y  o  > 
reen  and  choose  add  it 


cs  .  l  prc  t  o  cy  pc  c c r r  t  *  c  n  z  r. s  w  s  s  t,  * .  c.-  c  r.  * opt 
previously  chosen  in  the  program  setup.  This 
tc  choose  additional  ar.alvses  based  on  the  res 
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is  f  c  r  the  entire  £  re;?  :  f  s  u  b .;  e  c  t  s  ,  the  following  tetters 
exist : 

1.  Print  susiary  -  This  will  print  a  hardcopy  cf 
just  the  suniary  of  the  independence  axi ois. 

2 .  Print  ail  axiocs  -  This  will  print  a  haric:?y  cf 
the  results  of  the  entire  set  of  axiom  t e s t s , 
including  independence,  joint  independence,  and 
double  cancellation. 

3.  View  all  axioms  -  Allows  you  to  view  on  screen 
the  results  of  the  entire  set  cf  axiom  tests.  To 
scroll  through  all  of  the  results,  use  the  <UP  ARROW > 
and  <DGWN  ARROW>  keys.  When  finished  viewing  the 
entire  results,  the  <-->  key  will  return  you  to  this 
option  screen. 

4.  Proceed  To  Next  Option  Chosen  In  Program  Setup  - 
This  option  will  display  the  scaling  solution  for  the 
group  of  subjects,  if  scaling  was  chosen  in  program 
setup.  If  scaling  solution  was  not  chosen  the 
program  will  go  back  to  the  Program  Setup  screen. 

5 .  Return  To  Program  Setup  -  As  in.  the  previous 
screen,  this  option  allows  you  tc  return  to  the 
program  setup  and  change  the  ccnf i gur a t i on  cf  the 
analysis.  Other  options  previously  chosen  will  re 
cancelled,  and  new  ones  can  be  selected. 

6 .  Escape  -  As  before,  this  returns  you  to  the  main 
menu  ir.  case  cf  problems  cr  for  a  fast  way  tc  exit 
the  program. 
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If  the  Axiom  Test  Summary  is  for  a  prototype  group,  the  following 
options  exist: 


1.  Print  summary  -  same  as  for  group. 

2.  Print  all  asioms  -  same  as  for  group. 

0 .  View  all  axioms  -  same  as  for  group. 

4.  Proceed  to  next  option  chosen  in  program  setup  -  will 
display  the  scaling  solution  for  this  same  prototype  group, 
if  scaling  was  chosen  in  the  program  setup  screen. 

5.  Return  to  program  setup  -  same  as  for  group. 

6.  Go  to  next  prototype  -  will  display  summary  information 
of  axioms  violations  for  the  next  prototype  chosen  in  the 
program  setup.  Since  the  prototype  is  a  result  of  the 
correlation  matrix,  the  program  is  aware  of  the  next 


prototype  group. 

If  the  summary  is  for  an  individual,  the  following  options  exist 

1.  Print  summary  -  same  as  for  a  group. 

2.  Print  all  axioms  -  same  as  for  a  group. 

3.  View  all  axioms  -  same  as  for  a  group. 

4.  Proceed  Tc  Next  Option  Chosen  In  Program  Setup  -  same  as 
for  a  group. 

5.  Return  to  Program  Setup  -  same  as  for  a  group. 

G  .  Gc  to  Next  Individual  -  Will  display  summary  information 
of  independence  axiom  violations  for  the  next  individual 
chosen  in  the  program  setup. 


In  all  three  cases,  when  "Proceed  To  Next  Option  Chosen 
is  selected  on  the  Summary  of  Axiom  Violations  screen  and  a 
scaling  solution  option  had  been  chosen  on  the  Program  Setup 
Screen  or  upon  selection  of  the  "Scaling”  option  on  the  Main 
Menu  screen,  the  program  will  display  the  following 
information  on  the  Scaling  Information  Screen: 

The  last  five  iterations  of  the  Theta  and  Tau  values  for  the 
scaling  solution.  For  a  more  detailed  explanation  of  these 
values,  refer  to  the  section  "Interpretation  of  Output”, 

The  rescaled  values  for  each  level  of  the  subscales.  These 
are  the  additive  values  which,  through  all  possible 
combinations,  form  the  27  values  of  the  scaling  solution. 

The  approximate  relative  importance  of  each  factor.  This 
11*01  ■=,  u  e  s  *-  i  -  -  a  mi  -*  I-*  1  * 1 1  s  n  is  e  jr  r  o  r,\  levee  r  c  £  a  dime  n  s  i  o  n  c 
level  3  of  the  same  dimension. 
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TO  RUN  ANY  OF  THESE  PROGRAMS  OR  COMBINATIONS  OF  PROGRAM 
CHOOSE  THE  CORRESPONDING  NUMBER (S)  AND  PRESS  RETURN 

1  -  PROTOTYPE  CORRELATIONS  AND  KENDALL'S 

2  -  GROUP  AXIOMS 

3  -  GROUP  SCALE 

4  -  PROTOTYPE  AXIOMS 


5  -  PROTOTYPE  SCALE 


6  -  INDIVIDUAL  AXIOMS 


7  -  INDIVIDUAL  SCALES 


ESC  -  MAIN  MENU 
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WORKING . . 
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ASSUMPTION  1: 

WORKLOAD  IS  AN  IMPORTANT,  PRACTICALLY  RELEVANT,  MEASURABLE 
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SUBJECTIVE  WORKLOAD  ASSESSMENT  PROCEDURES: 

PROBLEMS 


SCALING  PROBLEMS  (ANCHORS,  INCREMENTS,  FORMAT) 


ASSUMPTION  2: 
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.  EVERYONE  KNOWS  WHAT 
WORKLOAD  IS.  RIGHT? 


CONCEPTUAL  FRAMEWORK  FOR  THE  ANALYSIS  OF  WORKLOAD  AND  PERFORMANCE 


wwmxni 


102 


UJ 

o 

z 

< 

1 

2 

S 

0., 


Z 

o 


to 


u 

UJ 

(C 

5  £ 

I  s 

a  o  j 

a:  a  uj 
8}  <  cc 


CA 


o 

ui 


CO 

Ul 


< 

a: 


to 

u. 

O 

Z 

o 


o 

UJ 


* 

(O  V, 

c  -i 

O  d 

*r 

O  to 
2  UJ 
>  > 
a:  y- 

O  5 
00  Z 

2  s 

ui  O 
to  u 


co 

UI 

O 

ec 

3 

O 


UI 

CO 

UJ 

to 

< 

cc 

CD 

u. 

UJ 

O 

O 

J— 

o 

Z 

-i 

UI 

UI 

< 

-l 

_J 

S 

y 

< 

S 

►- 

to 

>■ 

z 

a. 

►- 

o 

z 

SC 

1 

2 
O 

Z 

UJ 

2 

to 


Z 

o 


to  = 

_J  UJ 


s  c 

O  c 
o  u 


e 
3 
Q 

<  UI  u 
CC  I-  Q 
D  <  B 
O  ec  £ 


UJ 

_l 

CD 

< 

5 

cH 

Ui 

> 

K 

. w 
d  s 
<  3 
£  *“ 

s 

< 

o 

O  O 

us 

> 

UJ 

%  3 

K 

* 

s 

s  cc 

UJ  f- 

c/» 

> 

£ 

o 

1-  to 

to 

I 


jMk, 

*- 


V\\\ 


LU 

X 

O 


CO 

LU 


o 

h 


CL 


C/D 

C/D 

< 


co 

X 

O 

H 

O 

< 

Li¬ 


lli 

Z 

o 

X 


o 

o 


LL 

O 


CO 

LU 

Q 

3 

H 

Z 

CD 

< 


LU 

H 


> 

LL 


< 

3 

O 


< 

o 


CO 

x 

LU 


< 

CE 


2 

< 

LU 

o 

CO 

in 

CD 

H 

3 

Z 

CO 

< 

LU 

o 

3 

X 

>- 

LU 

X 

h- 

3 

3 

H- 

2 

Z 

< 

X 

h- 

•  • 

LU 

Q 

3 

3 

> 

o 

O 

3 

LU 

i — 

Z 

H 

< 

X 

Q 

< 

o 

h- 

CO 

3 

3 

< 

O 

o 

_ 1 

LU 
_ 1 

O 

3 

CL 

< 

X 

1 

< 

o 

CO 

CD 

Q 

Z 

< 

o 

£ 

> 

3 

X 

O 

z 

I- 

h* 

I"" 

< 

Z 

X 

X 

LU 

H* 

CO 

LU 

O 

z 

LU 

> 

CO 

o 

z 

o 

o 

o 

LU 

o 

3 

< 

”3 

CD 

LU 

CD 

3 

X 

o 

CO 

o 

3 

< 

2 

CD 

104 


PHASE  I 


_l 


3  Lt- 

Q  O 


waasm 


fttUMS  or 
wCLl-UiNC 


PWOM  of 
TASK 

v - x 

-/otlVATION 

/ 

/\ 

^INVI  HOWWCNT^ 

\ 

/ 

nto «ack  N 

V  ^ 

•J,' 


107 


MS 


C» 


\v 


CO 

2 

O 

t 

<  -7 


I — 


LL 
LL1 
CO  o 


a : 

o 

i — 

o 

< 

Ul 

LU 
X 
1 — 


Q 

< 

O 

X 

o 

$ 


CD  ^ 

"»  — j 
< 


h- 

< 


< 

> 

LU 

LL 

O 

Q 

O 


Q 

> 

a 


Q 

LU 

Q 


H  Q 
LU  ^ 


LU 

X 

< 


I 

o 

5 


o 

z 

UJ 

CD 

I 


UJ 

$ 

LL 

O 


©  UJ 

5  i 

UJ  H 
LU  < 


UJ 

O 

z 

< 

2 

x 

o 

u_ 

a; 

UJ 

o. 


UJ 

> 

CD 

X 

O 

< 

CO 

o 

UJ 

UJ 

LL 


CO 

CO 

UJ 

x 

H* 

co 


z  § 

8  § 

O  X 

2  o 

uj  m 


»- 

z 

UJ 

2 

Z 

o 

X 

> 

z 

UJ 


-j 

UJ 

> 

Ul 

—I 

0 


< 

cc 

I- 


O 

H 

< 

> 

H 

O 

2 


5 

CO 

UJ 

z 

> 

CO 

m 


< 

H 

UJ 


O 

2 

< 

i 

h- 

X 

o 

LL 

LL 

UJ 


CO 

CO 

Ul 

z 

> 

co 

3 

CD 


< 

O 

55 

> 


2  GL 


UJ 

3 

UJ 

CL 

o 

X 

>- 

S- 

H 

1 

•< 

1 

K 

GO 

CO 

CO 

CO 

UJ 

X 

UJ 

Q 

a 

CO 

UJ 

X 

X 

z 

z 

o 

-J 

CL 

< 

2 

UJ 

< 

2 

UJ 

a. 

tr 

CL 

2 

o 

0 

Z 

UJ 

UJ 

2 

a 

a 

CL 

a 

-J 

1- 

i 

i 

1 

i 

x 

x 

x 

X 

X 

X 

CO 

CO 

CO 

CO 

CO 

CO 

< 

< 

< 

< 

< 

< 

1- 

1- 

J— 

H 

h 

h- 

UJ 

CL 

> 

h“ 

I 

h- 

CC 

o 

£ 

UJ 


Ul 

Q 
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SINGLE  PILOT  ADVANCED  COCKPIT 
ENGINEERING  SIMULATION 


MISSION  SCENARIO  PROFILE 


O 


□  START  AND  END  WAYPOINT 
NAVIGATION 

O  PREDETERMINED  HOVER/FIRE  POINT 
J1  HOVER  AND  BOB-UP  MANEUVER 

AIR  TO  AIR  ENGAGEMENT 
-  ELAPSED  TIME/OR  DISTANCE 


OBJECTIVE 

•  INVESTIGATE  SINGLE-PILOT  OPERATION  IN  THE  NAP  OF  THE  EARTH  (NOE), 
FLIGHT  COMBAT  ENVIRONMENT 


APPROACH 

OBTAIN  PILOT  WORKLOAD,  PERFORMANCE,  AND  HANDLING  QUALITY 
RATINGS  FOR  DUAL  AND  SINGLE  PILOT  OPERATIONS 

•  SIMULATOR  •  AMES  VERTICAL  MOTION  SIMULATOR 

•  AIRCRAFT  MODEL  -  UH60  HELICOPTER 

•  COCKPIT  CONTROLLERS  -  2  ♦  1  ♦  1  * 

•  ADVANCED  DIGITAL  OPTICAL  CONTROL  SYSTEM  (ADOCS) 

-  ATTITUDE  COMMAND/STABILIZATION  OR  HYBRID  CONTROL  SYSTEM 

-  SELECTABLE  HEADING,  ALTITUDE,  POSITION,  AIRSPEED  HOLD.  TURN 
COORDINATION 

•  MISSION  MANAGEMENT  AND  COMMUNICATIONS 

RESULTS 

•  HIGH  WORKLOAD  FOR  ONE  PILOT 

•  ONLY  ONE  CONFIGURATION  SATISFACTORY  FOR  NOE  (ALTITUDE  HOLD  CRITICAL) 

•  HOVER  HOLD  ESSENTIAL  FOR  SINGLE  PILOT  OPERATIONS 

•  MISSION  MANAGEMENT  TASKS  DEGRADED  HANDLING  QUALITY  RATINGS 


COMPLEMENT  COMPARISON 


GHT  SEGMENT:  HQRs  and  WORKLOAD  RATINGS 


GURAT 


PHASE  IV 
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Correlations  among  raw  bipolar  ratings  and  WWL  scores 
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SUMMARY  OF  NASA  BIPOLAR  RATING  SCALE  EVALUATION 
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THEY  WERE  SUBDIVIDED 


/V 


OP,  EF,  FR  Subjective  responses/evaluations  of  behavioral  responses 
w  Subjective  weighting  of  factors 

Ewl  Integrated  subjective  experience  of  workload 

Rwl  Formal  numeric  or  verbal  evaluation  of  workload 


SELECTION  OF  SUBSCALES: 
NASA-TASK  LOAD  INDEX 
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OVERALL  WORKLOAD 
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SAMPLE  APPLICATION  OF  NASA  WORKLOAD  SCALE 

EXAMPLE: 

COMPARE  WORKLOAD  OP  TWO  TASKS  THAT  REQUIRE  A  SERIES  OF  DISCRETE 
RESPONSES.  THE  PRIMARY  DIFFICULTY  MANIPULATION  IS  THE  INTER-STIMULUS 
INTERVAL  IISI)  -  (TASK  1  «  500  risk.  TASK  2  -  300  msec) 


PAIR  WISE  COMPARISONS  OF  FACTORS: 

INSTRUCTIONS:  SELECT  THE  MEMBER  OF  EACH  PAIR  THAT  PROVIDED  THE  MOST 
SIGNIFICANT  SOURCE  OF  WORKLOAD  VARIATION  IN  THESE  TASKS 


TALLY  OF  IMPORTANCE 
SELECTIONS 

MD  ill  -  3 
PD  -  0 
TO  1 1 1 1 1  -  5 
OP  I  -  1 
FR  III  ■  3 

EF  III  »  3 

SUM  >  15 


RATING  SCALES: 

INSTRUCTIONS:  PLACE  A  MARK  ON  EACH  SCALE  THAT  REPRESENTS  THE  MAGNI 
TUOE  OF  EACH  FACTOR  IN  THE  TASK  YOU  JUST  PERFORMED 


DEMANDS 

RATINGS  FOR  TASK  1: 

RATING 

WEIGHT 

PRODUCT 

MD 

LOW 

x  1 

HIGH 

30 

X 

3 

90 

PD 

LOW 

X  1 

HIGH 

15 

X 

0 

0 

TD 

LOW 

x  1 

HIGH 

60 

X 

5 

150 

OP 

EXCL 

X  1 

POOR 

40 

X 

1 

40 

FR 

LOW 

1  _ 

x  1 

HIGH 

30 

X 

3 

90 

EF 

LOW 

X  1 

HIGH 

40 

X 

3 

120 

SUM 

490 

WEIGHTS  (TOTAL) 

15 

MEAN  WWL  SCORE 

32 

DEMANDS 

RATINGS  FOR  TASK  2 

RATING 

WEIGHT 

PRODUCT 

MD 

LOW 

1. 

X  1 

HIGH 

30 

X 

3 

90 

PD 

LOW 

1 

X  1 

HIGH 

25 

X 

0 

0 

TD 

LOW 

1 

x  ! 

HIGH 

70 

X 

5 

350 

OP 

EXCL 

1 

X  _  1 

POOR 

50 

X 

1 

50 

FR 

LOW 

1 

X  1 

HIGH 

50 

X 

3 

150 

EF 

LOW 

1 

X  1 

HIGH 

30 

X 

3 

90 

SUM 

730 

WEIGHTS  (TOTAL) 

15 

MEAN  WWL  SCORE 

49  ! 

RESULTS. 

SU8SCALES  PINPOINT  SPECIFIC  SOURCE  OF  WORKLOAD  VARIATION  BETWEEN 
TASKS  (TDl  THE  WWL  SCORE  REFLECTS  THE  IMPORTANCE  OF  THIS  AND  OTHER 
FACTORS  AS  WORKLOAD-DRIVERS  AND  THEIR  SUBJECTIVE  MAGNITUDE  IN 
EACH  TASK 


ULI 

X  z 


in 

Q 

b- 

< 

X 

Z 

cn 

z 

LO 

< 

z 

CO 

3 

7 

z 

o 

z 

X 

o 

a 

o 

• 

4— 1 

4-H 

i— i 

X 

X 

b- 

X 

< 

►- 

X 

in 

< 

< 

3 

cn 

CD 

t-4 

X 

UJ 

X 

4-4 

U 

< 

z 

z 

cn 

a: 

o 

X 

z 

4-4 

CD 

4-4 

< 

cn 

z 

O 

• 

_j 

> 

> 

*~ 

X 

O 

CJ 

CJ 

>- 

< 

< 

• 

CJ 

X 

cn 

X 

4-4 

CJ 

u 

X 

z 

> 

UJ 

1— 

< 

Z 

X 

z 

H- 

< 

4-4 

X 

< 

3 

X 

o 

o 

X 

'•X 

CD 

X 

o 

a 

X 

z 

X 

4-4 

4-* 

4-4 

Z 

3 

O 

z 

3 

4—4 

b- 

X 

4- 

X 

CJ 

_J 

• 

z 

UJ 

cn 

_1 

< 

CO 

< 

4- 

CJ 

o 

1 

X 

a 

< 

X 

4— 

4-4 

X 

< 

4—4 

> 

-J 

4— 

UJ 

IU 

CJ 

1-4 

< 

O 

X 

1- 

cn 

l-l 

-J 

z 

4-H 

H* 

4- 

X 

< 

< 

X 

> 

K- 

X 

f- 

< 

4-4 

b- 

cn 

X 

on 

X 

CJ 

22 

3 

X 

u 

< 

X 

-1 

_J 

X 

X 

O 

z 

X 

< 

< 

< 

H 

o 

“3 

UJ 

X 

X 

o 

v 

X 

3 

4— 

b~ 

X 

X 

a 

X 

X 

CD 

4-4 

CJ 

< 

z 

cn 

Z 

< 

cn 

z 

3 

h- 

3 

cn 

X 

4-H 

4-4 

UJ 

X 

X 

X 

< 

cn 

O 

UJ 

CD 

H* 

> 

z 

y 

X 

X 

>- 

X 

X 

X 

• 

z 

z 

X 

o 

< 

UJ 

(-4 

< 

H- 

X 

O 

o 

o 

o 

O 

O 

o 

o 

X 

u 

> 

z 

in 

V 

Ul 

UJ 

< 

UJ 

O 

CJ 

cn 

X 

z 

CD 

C_J 

z 

Ul 

4- 

X 

Z 

CJ 

< 

X 

o 

< 

< 

>: 

3 

UJ 

X 

X 

X 

cn 

3 

X 

o 

a 

o 

< 

4—4 

UJ 

< 

u 

UJ 

X 

• 

X 

X 

o 

X 

y 

3; 

cn 

-J 

UJ 

-j 

UJ 

> 

o 

X 

X 

u 

cn 

< 

X 

a 

»-* 

X 

a 

z 

a 

4— 

< 

X 

o 

id 

cn 

Z 

o 

X 

*=- 

cn 

> 

< 

z 

o 

_J 

3 

< 

4— 

4- 

UJ 

CJ 

V 

u 

4— 

UJ 

> 

z 

cn 

X 

UJ 

o 

>— 4 

b- 

Ul 

a 

-J 

> 

CD 

a 

z 

a 

X 

Z 

m 

*— i 

< 

X 

X 

z 

3 

4-4 

cn 

z 

< 

UJ 

»- 

3 

o 

X 

4-4 

UJ 

< 

> 

X 

X 

4H 

z 

X 

cn 

3 

4- 

Z 

X 

X 

UJ 

o 

cn 

< 

CD 

> 

X 

X 

4— 

< 

4-4 

UJ 

3 

UJ 

< 

u 

u 

4-t 

y 

Z 

X 

z 

X 

< 

X 

H" 

cn 

X 

x 

in 

3 

4-H 

o 

3 

X 

cn 

UJ 

< 

3 

z 

t-H 

> 

x. 

a 

cn 

HI 

y 

y 

cn 

X 

X 

X 

a 

UJ 

H- 

4-4 

Q 

b- 

in 

CJ 

3 

CJ 

X 

X 

< 

X 

> 

CD 

z 

< 

< 

4-4 

< 

< 

O 

< 

y 

X 

X 

V 

4-4 

z 

o 

X 

4— 

U 

_J 

H* 

3 

X 

4— 

z 

UJ 

4— * 

< 

b- 

UJ 

X 

O 

4-H 

H- 

Z 

4- 

u 

1 

4- 

CJ 

ic 

< 

X 

o 

CJ 

-J 

X 

4-H 

4-4 

V— 

.J 

X 

cj 

CJ 

UJ 

< 

< 

CD 

4- 

X 

z 

X 

< 

> 

3 

,  , 

4-t 

UJ 

CJ 

u 

4— 

X 

Z 

X 

o 

o 

4— 

a 

X 

z 

UJ 

CD 

H* 

,  , 

z 

z 

< 

o 

4— 

CJ 

X 

z 

a 

3 

< 

> 

O 

X 

< 

X 

Ul 

4- 

CJ 

cn 

4—4 

cn 

X 

X 

cn 

4-4 

1— 1 

X 

> 

CJ 

CJ 

z 

a 

►** 

a 

a 

< 

X 

X 

4-4 

< 

a 

t— t 

a 

o 

a 

< 

X 

O 

a 

o 

o 

o 

X 

o 

o 

V 

cn 

X 

X 

u 

Ul 

1  <  1 

X 

X 

"3 

> 

X 

X 

cr. 

X 

X 

*— 4 

X 

_ 

z 

X 

UJ 

UJ 

X 

X 

cj 

o 

X 

4-4 

< 

X 

X 

UJ 

X 

TOg 
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ABSTRACT 


This  report  presents  a  methodology  developed  to  measure  task 
performance  in  a  constrained  environment  Ci.e.,  Individual 
Protective  Equipment  (IPE)l.  The  methodology  has  potential  use 
in  a  number  ot  human  performance  measurement  areas  invol ving 
increased  time  to  complete  tasks  as  a  -function  of  changes  in  the 
usual  job  environment.  With  moderate  adjustments  to  an  algorithm 
developed  for  this  study,  performance  anal ysts  can  adapt  this 
method  to  cal cul ate  time  changes  for  other  degraded  job/task 
environments  which  exhibit  similar  character i st i cs  ,  such  as 
performance  changes  following  the  administration  of  pretreatment 
or  antidotal  drugs. 


INTRODUCTION 


BACKGROUND 


The  individual  protective  equipment  (IPE  )  is 
part  of  the  protective  posture  assumed  to  facil itate 
of  operations  in  a  chemically  toxic  environment,  and 
a  possible  cause  of  job  environment  degradation 
produce  substantial  increases  in  task  completion  time, 
of  tne  1 iterature  pertaining  to  human  performance  in  a 


an  integral 
conti nuance 
is  therefore 
wnich  might 
A  review 
chemical 


environment  revealed  that  the  information  is  not  sufficiently 
Quantified  to  provide  reliable  expectations  of  task  time 
increases  .  In  the  past,  the  procedure  used  to  measure  task  time 
increase  has  been  to:  1.  observe  tasks  being  performed  by 
individuals  wearing  the  IPE;  2.  record  the  amount  of  time 
required  to  accompl  ish  the  task.  :  3.  compare  that  time  to  a 

basel ine  measure  which  is  typical ly  the  tasl  time  performance  in 
a  shirt-sleeve  environment;  4.  define  the  difference  between  the 
shirt-sleeve  time  and  IFE  time  as  the  increase  in  time  to 
perform.  CCcx  and  Jeffers  ( 1 98 1 )  ;  Hinch  (1982)1.  These  studies 
along  with  other  similar  field  studies  (currently  being  performed 
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bv  JAYCOR)  ,  are  the  beginning  of  a  data  base  that  may  be  used  by 
simulation  modelers. 

t.  s 

•/./: 

I  PE  COMPONENTS 

The  IPE  worn  by  Air  Force  ground  support  personnel  consists 
o-f  the  -following  pieces  o-f  equipment:  M17A1  protective  mask,  M6A2 
protective  hood,  butyl  rubber  gloves  with  cotton  liners, 
overgarment  consisting  o-f  jacket  and  pants  with  charcoal 
interlining  worn  over  fatigues,  and  overboots.  Although  the  IPE 
provides  protection  for  personnel  in  a  toxic  environment,  tne 
drawbacks  of  this  protection  are  heat  stress,  encumbrance, 
degraded  dexterity,  restricted  vision  and  lowered  communication 
ability.  As  a  result,  the  time  required  to  complete  many 
maintenance  tasks  is  increased  ,  and  the  capability  to  complete 
some  tasks  is  seriously  impaired. 

SCOPE  OF  STUDY 

This  investigation  was  initiated  specifically  to  bevel oo  a 
metnccol ogv  for  determining  the  task  time  increase  for  aircraft 
maintenance  and  munitions  tasks  as  thev  apDl y  to  the  Chemical 
Warfare  Theater  Simulation  of  Airbase  Resources  (CWTSAR)  comouter 
model  . 


The  scope  of  this  studv  included  the  following  objectives 


Perform  a  review  of  the  literature  concerning  measurement  o~ 
human  performance  in  a  degraded  environment. 

Determine  which  human  ability  factors  and  task  character i st ics 
might  best  represent  the  components  in  a  human  performance 
model  . 

Develop  a  methodology  for  calculating  the  increase  in  time 
required  to  complete  aircraft  maintenance  tasks  while  wearing 
I  PE  . 

Determine  task  time  multipliers  that  estimate  the  time 
increase  to  complete  various  aircraft  maintenance  tas(  s  while 
wearing  the  IPE. 

Implement  a  human  performance  data  base  for  use  bv 
simulation  modelers  and  jod  performance  analysts. 


A  number  of  potentially  interesting  factors  that  were  no* 
considered  in  the  method  developed  in  this  paper  were: 


o  Amo  lent  temperature, 
c  Work /rest  cycles. 

o  Safety  factors  in  a  degraded  environment, 
o  Si  1 1  1  levels  of  individuals  performing  tne  jobs. 


These 


var i ab 1 es 


performance  data 


mav  eventually  become  part 
base,  but  are  not  considered 


of  the  n uman 
in  the  scope  of 


this  studv.  A  short  discussion  of  the  human  performance 
the  humor,  ab;1  ities,  and  t-sk  a...- —or'  selected  to  sup;-: 
data  base  is  presented  in  tha  following  section. 


mcd  e  1 


■»:' 


•A 


a 


a 


mA  uSW  -V  JSi  -V  kN  Al  m  *^*  - *-“-*»■  *  «  »  «-  *  -  A  »  W  A  ■A  JS.  *2  *  a,*  m  .JV 


isVisnvi 


HUMAN  performance  model 

The  human  oertormance  model  shown  -n  Figure  1  graphically 
displays  "someone,  doing  something,  somewhere"  [Bailey  (1982)], 
but  in  this  case  the  someone.  something,  and  somewhere  are 
constrained  bv  the  chemical  war-fare  environment  in  which  the 
tasks  are  being  performed . 


Figure  1  Human  Pert 


Model 


This  illustration  presents  the  reader  with  an  example  of  the 
man -machine  inter* ace  in  a  normal  environmental  setting  (human, 
environment.  tasks  to  be  performed >  .  wnich  the  data  base 
represents  as  a  oasei  me  fcr  tasl  completion  and  human 
abilities.  Bailey's  model  was  then  enclosed  in  constraints.  The 
constraints  are  represented  in  the  data  base  as  difficulty 
factors.  A  1  ike! v  consecusnce  of  increased  difficulty  is  an 
increase  in  performance  time. 


HUMAN  AB  I  L I T I E  a  Anil1  T  Abi  ['Er  ill! 

It  is  the  premise  of  this  ecu c/  that  human  ad i 1 ities  can  be 
rated  according  to  tne  1  eve)  o*  importance  or  "criticality"  when 
performing  fa  si  s.  The  eight  ‘“..man  ao : )  it-  categories  and  them 


c  o  r  •*  e  s  d  o  n  a  :ru 


e  g  o  r  i  e  s  w  e  -  e  o  &  v  e 


> a  s  e  o  on  the 


ta..or.om.  and  factor  analytic  wori  performed  c  ,■  Fleishman  ■.  1-^62  . 
i'°72.  i  9  7*5 .  1*78.  and  1982''  .  these  ap  i  1  ities  are  assumed  to 

adeou-te;  v  mar  act  :  :  e  the  demands  of  tael  ?  -rat  are  mitical  to 
a  mil  if;--.-  ma  ir  ter.  an  re  en  .<  m  c  -men  t  .  fr-,e  eight  ab  1 1  ities  •.  r  ea¬ 
table  1  >  describe  .-a;.  -  function  ;  reeced  ft  :e-:r  .  continte.  a- t 


As  discussed  b  -  F'eisnmar.  ■  :  5  7  •  .  me  term  "ab  1  1  it  /“  re*  e-< 

to  a  general  *:ra:*;  wnich  has  peer  infer*  eo  *'•3*  consister- 
responses  on  certain  fasi  s.  The  human  performance  ab 1 1  it 
categories  m  Tapie  i  form  the  basis  *or  ftp  subjective  ratin' 
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fflV 


t i me  mu 1 t 1 p 1 l ers  ( TTM ) . 

A  task,  as  defined  in  this  study,  is  one  of  the  major 
subdivisions  o-f  a  job  performed  by  an  individual  .  A  job  usually 
consists  o-f  between  two  and  seven  tasks.  The  following  are  some 
character i st l cs  of  a  task: 

o  It  is  one  of  the  mam  functions  required  for  the  personnel  to 
perform  in  order  to  complete  the  job  successful  1 y . 
o  It  comprises  a  grouping  of  closely  rel ated  subtasks, 
o  Task  requirements  often  are  the  basis  for  initial  assignment  to 
a  job,  and  for  determining  the  qualifications  required  to 
perform  the  job . 

TABLE  ONE.  HUNAN  PERFORMANCE  ABILITY  CATEGORIES 


Audieory 

Decection 


Dexterity 


Psychological 

Effects 


localization 
sensicivicy 
response  rate 
soeech  interference 
intensity 


fine  motor 
manipulation 
fine  motor 
response 
fine  motor 
strength 


tension  • 

depression 

anxiety 

confusion 

motivation 


Cognitive 

Effects 


Vision 


shore  term 
long  term 
retention 
storage 
concent  rac ion 
attention 


acuity 

accommodation 

distance 

perception 

color  discrimination 
peripheral  vision 


Communicac ion 


Phys lo logics  L 
Cor.d  it  ions 


Physical 
Coord  mat  ion 


understand  speech 
resnonse  process 


f  at  igue 


adapt  at  ion 


motor  response 
general  mobility 
strength 


A  practical  method  tor  estimating  mere 
performance  time  in  a  degreced  environment  can 
subjective  estimates  of  tne  critics'  itv  of  hu 
'Gephardt,  et.  at  ..  1  3  3 1  1  to  t  a  st  oer  r  or  mane  e  .  It  i 
tnat  increases  in  a  i  r  f  j  i  t  v  -les  comoared  t 
en  v  i  e'onmen  t  >  due  to  wesrina  IFF  can  be  subjective! 
job  incumcents.  The  present  method  obtains  Subject 
of  critical  it v  anc  m creased  difficult-.-  <  using 
scale  for  botn  measures  1  tor  each  of  the  human  ab i ! 
shown  in  Table  1.  Trese  estimates  ?re  then  combined 
cel ow)  in  order  to  obtain  an  overal I  Tasi  Time  Mult 
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A  f 1 ow  d 
devel opment 
reader  with 
the  task  time 


lagram  (Figure  2)  displays  the  process 
of  the  data  base.  This  diagram  also 
an  overview  o-f  the  steps  which  lead  to 
mul  tipi  ler  al  gonthm  . 


emp 1 oyed  f  or 
provides  the 
the  design  o-f 


( - 'N 


Figure  Two  Flow  Diagram 


I h  e  eight 
represen  tat i ve 


human  abilities  tentatively  identified  as 
of  the  components  required  for  task  performance 
included  in  this  data  base  are  listed  in  Table  2  with  their 
respective  definitions.  It  may  appear  that  parts  of  definitions 
overl  e.D  or  fit  into  other  categories.  but  until  more  research 
dictates  further  refinement,  these  definitions  will  be  used. 


TABLE  2  Ab i 1  1 1 v  Def i n 1 1 1 on i 


l.  Auditory  drtrcdw  THe  aenettlvlty  to  eownda  (other  than  vrrbil 

tawmiiftloii) .  and  lnterpratat  loa  of 
touidi . 


2.  Cognidvr  effect* 


3.  CoMunlcxlan 


Ability  to  concentrate  and  atcand  to  a  task. 
Inclidta  th«  abort-taro  and  long-term 
oaoory.  ratantlon,  and  recall  o( 

Information. 

Ability  t>  under  a tend  and  respond  CO  speech 
(or  relating  Idaaa  naettaary  (or  task 
comp  let Ion. 


4.  Devterlty 


5.  Physical  coordination 


Fln«  notor  response,  (Inn  aocor  Manipulation 
(ability  to  vork  by  touch  wK»«  unable  to  ««« 
an  object),  (ln«  Motor  strength. 

The  ability  to  climb,  drive,  walk,  uaa 
general  mobility,  and  miacular  atrength. 


b.  Phya io loglca 1  conditions 


Tactile  pressure ,  fatigue,  personal  needs, 
and  aramlna  eapertencad  In  performance  o  ( 
tasks . 


f.  P  a  yc  he log  leal  effects 


Levels  of  stress,  tanalon.  depression, 
snalety.  confusion,  and  sotlveclon. 


(.  Vision 


Use  of  acuity,  accoiodst  Ion .  depth 
perception,  adaptation,  color  liter lalna- 
.  ion.  a-.d  pei'.pncral  <l*tun  co  perform  a 


HUMAN  PERFORMANCE  MATRIX 


A  representat ion  of  the  matrix  used  to  display  the  various 
•factors  is  shown  in  Figure  3.  The  matrix  displays,  as  an  example, 
vehicle  mechanic  tasks.  The  contrived  task  percentage,  human 
performance  abilities,  criticality  factors,  and  difficulty 
factors  have  been  arbitrarily  assigned.  These  values  would,  in 
theory,  yield  the  TTM  for  each  task  the  vehicle  mechanic 
performs.  A  discussion  of  each  section  of  the  matrix  beginning 
with  tne  task  percentage  is  presented  below. 
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tasks  that  comorise  the  particular  job  being  measured 
anted  across  the  top  of  Figure  3.  Uni  ike  tne  ab l 1  lties 


earl  i e1" ,  task  s  wil  1  change  according  to  the  job  being 
Task  specifications  can  be  found  i.n  references  such 
Force  Regulation  (AFR.)  39-1,  "Airman  Classification." 
e  of  cast  performance  is  defined  as  the  percent  of  the 
b  that  an  individual  so ends  performing  tne  task .  For 
the  tasks  of  a  vehicle  mechanic  are: 
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Fne  mechanic  may  soend  3iZ>  percent  -of  the  job  time  tuning  engines 
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and  5  percent  repairing  exhaust  systems.  It  is  assumed  that  most 
job  cl  assi-f  icat  ions  can  be  viewed  in  this  manner.  The  performance 
anal yst  may  evaluate  any  task  which  is  performed .  Those  tasks 
possessing  high  criticality  and  high  difficulty  factors,  may 
need  equipment  or  task  redesign  to  enhance  performance  in  a 
degraded  environment.  Conversely,  a  task  which  is  performed 
infrequently,  and  has  low  criticality  and  difficulty  factors,  is 
one  tnat  may  net  need  further  investigation. 

CRITICALITY  FACTOR 

The  criticality  factor  is  defined  as  the  level  of  importance 
of  a  human  ability  to  the  performance  of  a  task,  development  of 
this  factor  consists  of  establishment  of  the  relative 
importance  of  each  of  the  eight  abilities  to  completion  of  a 
task.  A  questionnaire  being  developed  by  JAYCOR  will  collect  the 
criticality  data.  This  questionnaire  will  be  distributed  to 
personnel  performing  the  tasks  and  they  will  subjectively  rate 
the  importance  of  each  factor  to  task  completion.  The 
questionnaire  uses  a  five  point  scale,  one  equal  to  not  important 
and  five  equal  to  very  important  . 

DIFFICULTY  FACTOR 

The  level  of  difficulty  is  the  amount  of  constraint  placed 
an  an  individual  by  changes  in  the  environment .  The  difficulty 
factor  is  a  function  not  only  of  the  task,  but  also  of  the 
particular  source  of  task  degradation  (IRE,  pretreatment  ,  etc.). 
The  procedure  for  determining  the  difficulty  factor  is  also  a 
subjective  rating  (one  equal  to  not  difficult  and  five  equal  to 
very  difficult)  obtained  from  the  personnel  performing  the  task. 

The  critical ity  factors  are  rated  in  order  to  establ ish  a 
weighting  factor  for  each  of  the  human  abilities.  The  difficulty 
ratings  are  obtained  following  task  completion  while  wearing 
the  IF'E.  It  is  not  necessary  to  have  the  same  individual  rate  the 
criticality  and  the  difficulty  factors  for  the  same  task.  Both 
the  criticality  and  difficulty  factors  are  measures  of  central 
tendency  (::)  for  a  number  of  individuals  and  therefore  are 

averages  of  a  set  of  data. 

TASK  TIKE  MULTIPLIER 

The  TTM  for  each  task  is  arrived  at  by  employing  the 

following  algorithm: 


o  For  each  of  the  eight  human  abilities,  the  criticality  factor 
is  multiplied  bv  the  difficulty  factor, 
o  These  products  are  summed  over  the  eight  human  abilities, 
o  The  sum  of  the  products  is  then  divided  by  the  sum  of  the 
eight  criticality  factors  to  give  the  performance  number, 
o  The  performance  number  is  then  converted  to  the  TTM,  bv 
1  inearl y  rescaling  its  range  of  possible  values,  1  to  5,  to 


the  TTM  range,  1  to  2.2  Ci.e.,  TTM  =  1  +  .97  (P.N.  -  1)3  using 
the  conversion  scale  discussed  below. 


CONVERSION  SCALE  DEVELOPMENT 

The  conversion  scale,  used  in  the_ above  algorithm,  provides 
the  transition  -from  the  Performance  Number  to  the  TTM.  This 
scale,  as  displayed  at  the  bottom  of  Figure  3  allows  for  the 
1  inear  rescal  ing  of  the  performance  number  to  the  task  time 
multiplier.  This  scale  has  been  developed  from  operational  field 
data  pertaining  to  munitions  and  maintenance  tasks.  Eighty-five 
task  times  were  compiled  from  reports  of  operational  chemical 
defense  training  exercises.  Of  these  85  tasks  measured  when 
wearing  the  IPE,  only  one  task  time  exceeded  twice  the  normal 
completion  time.  Based  on  these  data,  it  was  empirically 
determined  to  set  the  highest  value  of  the  conversion  scale  at 
2.2.  Therefore,  if  tuning  an  engine  takes  80  minutes  and  is 
increased  by  a  TTM  of  1.97,  then  the  increased  task  time  would  be 
157.5  minutes.  This  scale  is  in  the  process  of  being  refined  to 
associate  the  performance  number  to  a  TTM  more  accurately.  As 
additional  data  become  available,  the  conversion  seal e  will  be 
adjusted  to  reflect  any  changes  . 


CONCLUSIONS  AND  RECOMMENDATIONS 


CONCLUSIONS 

The  conclusions  of  this  study  are: 

o  Human  abilities  and  task  characteristics  for  a  human 
performance  model  can  be  roughly  defined. 

o  A  methodol ogv  for  calculating  the  increased  time  for  tasks  in 
a  degraded  environment  has  been  developed. 

It  is  further  concluded  that: 

o  Data  collection  for  various  aircraft  maintenance  tasks  is 
required  in  order  to  develop  a  more  comprehensive  data  base 
and  to  test  various  simulation  models. 

o  Implementation  of  a  human  performance  data  base  for  use  with 
simulation  models  is  of  value  and  enhances  the  utility  of  the 
mode  1 s  . 

The  human  performance  data  base  developed  through  this  study 

provides  the  following: 


o  A  method  of  calculating  increased  time  to  perform  the  task. 


o  Ident  i-f  icat  ion  of  tasks  within  the  job  which  may  exceed 
reasonable  time  to  complete. 

o  Identification  of  human  abilities  that  are  being  stressed  and 
that  may  be  high  contributors  to  task  time  increase. 

o  Tasks  that  may  need  further  evaluation  or  the  redesign  of 
equipment  as  a  method  of  decrease  task  time. 


RECOMMENDATIONS 

Although  progress  has  been  made,  there  is  still  considerable 
validity  and  reliability  testing  required.  As  data  are  collected 
and  added  to  the  data  base  the  strength  of  this  methodology  will 
be  tested.  The  following  recommendations  are  presented  as 
conditions  for  implementation  and  testing  of  the  human 
performance  data  base: 

o  Continue  to  solicit  opportunities  for  operational  field  data 
col  1 ect ion  . 

a  Implement  software  to  perform  data  runs  for  prel iminary 
validation  of  the  human  performance  data  base. 

o  Continue  refinement  of  the  conversion  scale  for  TTM 

rel  i ao 1 1  i tv  . 
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RETURN  TO: 

HARRY  G.  ARMSTRONG  AEROSPACE  MEDICAL  RESEARCH  LABORATORY 
SPECIAL  PROJECTS  OFFICE 
AEROSPACE  MEDICAL  DIVISION 
AIR  FORCE  SYSTEMS  COMMAND  (AAMRL/HET) 
WRIGHT-PATTERSON  AIR  FORCE  BASE,  OHIO  45433-6573 

ATTENTION:  2  LT.  MASAK 


Autovon:  785-7583 


Commercial:  (513)  255-7583 


BIOGRAPHICAL  DATA  SHEET 


Please  fill  out  the  biographical  data  sheet  as  accurately  and  completely  as 
possible.  All  of  your  comments  will  be  kept  confidential. 


I. 


DATE:  _ 

MILITARY  INFORMATION: 

1.  AFSC  No.:  _  2.  Job  Title  _ 

3.  Rank:  _  4.  Duty  Phone:  _ 

5.  Length  of  Time  in  Active  Military  Duty:  _ 

_  years  and  _  months 

6.  Number  of  Chemical  Warfare  Exercises  You  Have  Participated  in 


7.  Number  of  Chemical  Warfare  Exercises  You  Have  Participated  in 

during  the  Past  Year  _ 

8.  Number  of  Months  You  Have  Been  Stationed  at  this  Base 


II.  PHYSICAL  INFORMATION: 


1. 

Age: 

(years) 

2. 

Height: 

(feet) , 

(inches) 

3. 

Weight : 

(pounds) 

4. 

Sex: 

Male 

Female 

III.  Work  Conditions 


During  combat  conditions  where  would  you  perform  your  job? 

a.  Inside 

b.  Outside 

c.  Both 


Explain 


•» 

* 
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HUMAN  PERFORMANCE  QUESTIONNAIRE 


PURPOSE 


The  purpose  of  this  questionnaire  is  to  gather  information  about  various 
aspects  of  your  AFSC.  The  data  collected  will  be  used  to  determine  how 
your  job  performance  might  be  affected  by  such  factors  as  wearing  the 
chemical  defense  ensemble,  and  other  changes  in  your  work  environment. 


INSTRUCTIONS 


1.  To  complete  the  questionnaire,  circle  the  appropriate  number 
corresponding  to  your  answer.  For  example: 

How  important  is  clear  vision  to  the  performance  of  your  job? 

1  Not  important 

2  Somewhat  important 

3  Important 

(3J)  Very  important 
5  Most  important 

2.  Again,  all  responses  will  be  confidential. 

3.  If  you  have  any  questions,  please  ask  the  interviewer. 
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1.  How  important  is  clear  vision  to  the  performance  of  your  job? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

2.  How  important  is  the  ability  to  quickly  focus  your  eyes  (up  close  and 
then  far  or  far  then  up  close)  to  the  performance  of  your  job? 

1.  Not  important 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

*  5.  Most  important. 

3.  How  important  is  the  ability  to  see  objects  at  a  distance? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

4.  How  important  to  your- job*  is  depth  perception  (three-dimensional)? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

.  4.  Very  important. 

5.  Most  important. 

5.  How  important  is  it  for  your  eyes  to  adjust  to  varying  amounts  of  light 
(brightness/darkness) ? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  Important. 

5.  Most  important. 


*<• 


How  important  to  the  performance  of  your  job  is  the  ability  to 
distinguish  different  colors? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

How  important  to  the  performance  of  your  job  is  the  ability  to  watch 
monitor  instruments  or  gages? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

Overall ,  how  important  is  vision  to  the  performance  of  your  job? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

A.  Very  important. 

5.  Most  important. 

How  important  to  your  job  is  the  ability  to  hear  emergency  sounds? 
(List  specific  emergency  sounds.) 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

A.  Very  important. 

5.  Most  important. 
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10.  How  important  to  your  job  is  the  ability  to  determine  volume  levels 
(sound) ? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

11.  How  important  to  your  performance  are  workplace  noise  levels? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

12.  How  important  is  sensitive  hearing  to  the  performance  of  your  job 
(e.g.,  audio  equipment)? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

13.  How  important  is  the  ability  to  hear  equipment  sounds 
(e.g.,  troubleshooting  equipment  sounds)? 

1;  Not  important. 

2.  Somewhat  important.  . 

3.  Important. 

4.  Very  important. 

5.  Most  important. 
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14.  How  important  to  your  job  is  the  ability  to  clearly  hear  others 
speaking? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

15.  How  important  is  it  for  you  to  react  to  sounds? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

16.  Overall ,  how  important  is  hearing  to  the  performance  of  your  job? 

..  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

17.  How  important  is  a  sense  of  balance  to  the  performance  of  your- job? 

1.  Not  important. 

2.  Somewhat  important. 

3;  Important. 

4.  Very  important. 

5.  Most  important. 

18.  How  important  is  being  fully  rested  to  the  performance  of  your  job? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 
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19.  How  important  to  your  job  is  the  ability  to  do  physical  work  for  long 
periods  of  time? 


1.  Not  important. 


2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 


5.  Most  important. 


20.  Overall ,  how  important  is  your  physical  condition  to  the  performance  of 
your  jo*'? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 


5.  Most  important. 


How  important  to  your  job  is  climbing? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  Important. 

5.  Most  important. 


How  important  to  your  job  is  driving? 

1.  Not  important. 


2.  Somewhat  important. 

3.  Important. 


4.  Very  important. 


3.  Most  important. 


How  important  to  your  job  is  walking? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 
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28.  How  much  influence  does  depression  (e.g.,  sadness,  despair)  have  on  the 
performance  of  your  job? 

1.  Not  influenced. 

2.  Somewhat  influenced. 

3.  Influenced. 

4.  Very  influenced. 

5.  Most  influenced. 

29.  How  much  influence  does  anxiety  have  on  the  performance  of  your  job 
(e.g.,  worried,  concerned)? 

1.  Not  influenced. 

2.  Somewhat  influenced. 

3.  Influenced. 

4.  Very  influenced. 

5.  Most  influenced. 

30.  How  much  influence  does  feeling  confused  have  on  the  performance  of 
your  job  (e.g.,  not  understanding  directions  for  given  job)? 

1.  Not  influenced. 

2.  Somewhat  influenced. 

3.  Influenced. 

4.  Very  influenced. 

5.  Most  influenced. 

31.  How  much  influence  does  motivation  have  on  the  performance  of  your  job? 

1.  Not  influenced. 

2.  Somewhat  influenced. 

3.  Influenced. 

4.  Very  influenced. 

5.  Most  influenced. 


32.  How  important  is  the  ability  to  recall  procedures  (Tech  Orders)? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

33.  How  important  is  the  ability  to  quickly  recall  correct  procedures  to 
the  performance  of  your  job  (e.g.,  emergency,  safety  procedures)? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

34.  How  important  to  your  job  is  the  ability  to  concentrate  under  stress 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

35.  How  important  is  the  ability  to  focus  attention  to  the  jotr  •artr-haml? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

36.  How  important  is  the  ability  to  quickly  perform  logical  processes 
(e.g.,  math,  technical  procedures)? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 


37.  Overall ,  how  important  is  memory  to  the  performance  of  your  job? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

38.  How  important  to  your  job  performance  is  the  ability  to  understand  what 
someone  is  saying? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

39.  How  important  to  your  job  is  the  ability  to  speak  clearly? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

40.  How  important  to  your  job  is  the  ability  to  write  legibly? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

41.  Overall ,  how  important  to  your  job  is  the  ability  to  communicate? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 


d')  l  '%. «•  I 


205 


42.  How  important  is  dexterity  to  the  performance  of  your  job? 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

43.  Using  the  following  scale,  please  rate  each  of  the  items  as  it  pertains 
to  your  job: 

1.  Not  important. 

2.  Somewhat  important. 

3.  Important. 

4.  Very  important. 

5.  Most  important. 

_  Making  Keystrokes  (typewriter,  computer,  calculator) 

_  Pushing  Buttons,  knobs 

_  Flicking  Switches 

_  Using  Clamps 

_  Manipulating  Levers 

_  Using  Screwdrivers,  tools 

_  Turning  Dials 

_  Using  Keys 

_ _  Using  a  Firm  Grip 

_  Making  Steady  Wrist  Movements 

_  Aligning  Equipment 

_  Tracking  Objects  Across  a  Screen 

_  Inserting  Objects  or  Instruments  into  Small  Places 

_  Threading  Objects  (Stripped  Bolts,  Needles) 
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INSTRUCTIONS 


The  following  questions  are  designed  to  gather  information  about  the 
difficulties  you  may  have  experienced  while  performing  your  job  in  IPE.  T 
data  will  be  used  to  help  to  eliminate  some  of  these  difficulties.  To 
complete  the  questionnaire,  circle  the  appropriate  number  corresponding  to 
your  answer. 

How  difficult  is  it  to  see  clearly  while  wearing  IPE? 

1.  Not  difficult. 

©  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

If  you  have  any  questions,  please  ask  the  interviewer. 
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How  difficult  is  it  to  see  clearly  while  wearing  IPE? 


1.  Not  difficult. 


2.  Somewhat  difficult. 


3.  Difficult. 


4.  Very  difficult. 


5.  Most  difficult. 


2.  How  difficult  is  it  to  quickly  focus  your  eyes  (up  close  and  then  far 
or  far  then  up  close)  while  wearing  IPE? 


1.  Not  difficult 


2.  Somewhat  difficult. 


3.  Difficult. 


4.  Very  difficult. 


5.  MosC  difficult. 


How  difficult  is  it  to  see  objects  at  a  distance  while  wearing  IPE? 


1.  Not  difficult. 


2.  Somewhat  difficult. 


3.  Difficult. 


4.  Very  difficult. 


5.  Most  difficult. 


4.  How  difficult  is  it  to-use~depth~perception  (three-dimensional  vision) 
while  wearing  IPE? 


1.  Not  difficult, 


2.  Somewhat  difficult. 


3.  Difficult. 


4.  Very  difficult. 


5.  Most  difficult. 
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5.  How  difficult  is  it  for  your  eyes  to  adjust  to  varying  amounts  of  light 
(brightness/darkness)  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


6.  How  difficult  is  it  to  distinguish 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


different  colors  while  wearing  IPE? 


7.  How  difficult  is  it  to  watch  or  monitor  instruments  or  gages  while 
wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

8.  Overall,  how  difficult  is  it  to  see  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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9.  How  difficult  Is  it  to  hear  emergency  sounds  while  wearing  IPE?  (List 
specific  emergency  sounds.) 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

10.  How  difficult  is  it  to  determine  volume  levels  (sound)  while  wearing 
IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

11.  How  much  influence  do  work  place  noise  levels  have  on  the  performance 
of  your  job  while  wearing  IPE? 

1.  No  influence. 

2.  Some  influence. 

3.  Moderate  Influence. 

4.  High  influence. 

3.  Excessive  influence. 

12.  How  difficult  is  it  to  clearly  distinguish  different  sounds  while 
wearing  IPE  (e.g.,  audio  equipment)? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


13.  How  difficult  is  it  to  hear  equipment  sounds  (e.g.,  troubleshooting 
equipment  sounds)  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

14.  How  difficult  is  it  to  clearly  hear  others  speaking  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

13.  How  difficult  is  it  for  you  to  react  to  sounds  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

16.  Overall ,  how  difficult  is  it  to  hear  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

17.  While  wearing  IPE,  how  difficult  is  it  to  maintain  a  sense  of  balance? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


18.  How  difficult  is  it  to  ‘terform  your  job  in  the  IPE  while  feeling  tired? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


19.  How  difficult  is  it  to  do  physical  work  for  long  periods  of  time  while 
wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


20.  Overall .  how  difficult  would  it  be  to  perform  your  job  in  the  IPE  if  in 
poor  physical  condition? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


How  difficult  is  climbing  while  wearing  IPE? 

1.  Not  difficult. 

2v  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


How  difficult  is  driving  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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18.  How  difficult  is  it  to  perform  your  job  in  the  IPE  while  feeling  tired?  VV 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


19.  How  difficult  is  it  to  do  physical  work  for  long  periods  of  time  while 
wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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20.  Overall ,  how  difficult  would  it  be  to  perform  your  job  in  the  IPE  if  in 
poor  physical  condition? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


21. 


How  difficult  is  climbing  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


22. 


How  difficult  is  driving  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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23.  How  difficult  is  walking  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

24.  How  difficult  is  general  movement  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

25.  How  difficult  is  it  to  maintain  muscular  strength  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

26.  Overall,  how  difficult  is  it  to  maintain  good  physical  proficiency 
while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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The  following  five  questions  are  on  aspects  other  than 
difficulty. 


>> 


27.  How  much  stress  or  tension  is  caused  by  the  IPE? 

1.  No  stress. 

2.  Some  stress. 

3.  Moderate  stress. 

4.  Extreme  stress. 

5.  Severe  stress. 

28.  How  depressed  (e.g.,  sadness,  despair)  do  you  feel  while  wearing  IPE? 

1.  No  depression. 

2.  Some  depression. 

3.  Moderate  depression. 

4.  Extreme  depression. 

5.  Severe  depression. 

29.  How  much  anxiety  do  you  feel  while  wearing  IPE  (e.g.,  worried  or 
concerned)? 

1.  No  anxiety. 

2.  Some  anxiety. 

3.  Moderate  anxiety. 

4.  Extreme  anxiety. 

5.  Severe  anxiety. 

30.  How  much  confusion  do  you  feel  while  wearing  IPE  (e.g.,  not 
understanding  directions,  for  given  job)? 

1.  No  confusion. 

2.  Some  confusion. 

3.  Moderate  confusion. 

4.  Extreme  confusion. 

5.  Severe  confusion. 


A 


■to  s 

>;v  ^ 


20 


'A 


How  much  stress  or  tension  is  caused  by  the  IPE? 

1.  No  stress. 

2.  Some  stress. 

3.  Moderate  stress. 

4.  Extreme  stress. 

5.  Severe  stress. 


28.  How  depressed  (e.g.,  sadness,  despair)  do  you  feel  while  wearing  IPE? 

1.  No  depression. 

2.  Some  depression. 

3.  Moderate  depression. 

4.  Extreme  depression. 

5.  Severe  depression. 


29.  How  much  anxiety  do  you  feel  while  wearing  IPE  (e.g.,  worried  or 
concerned) ? 

1.  No  anxiety. 

2.  Some  anxiety. 

3.  Moderate  anxiety. 

4.  Extreme  anxiety. 

5.  Severe  anxiety. 

30.  How  much  confusion  do  you  feel  while  wearing  IPE  (e.g.-,-  not 
understanding  directions,  for  given  job)? 

1.  No  confusion. 

2.  Some  confusion. 

3.  Moderate  confusion. 

4.  Extreme  confusion. 

5.  Severe  confusion. 
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H^v  is  your  motivation  to  perform  your  job  well  affected  by  wearing 
IPE? 

1.  No  motivation. 

2.  Some  motivation. 

3.  Moderate  motivation. 

4.  Strong  motivation. 

5.  High  motivatio'n. 

How  difficult  is  it  to  recall  procedures  (Tech  Orders)  while  wearing 
IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

How  difficult  is  it  to  quickly  recall  correct  procedures  while  wearir 


How  difficult  is  it  to  quickly  recall  correct  procedures  while  wearing 
IPE  (e.g.,  emergency  safety  procedures)? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


34.  How  difficult  is  it  to  concentrate  under  stress  while  wearing  IPE? 
1-.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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35.  How  difficult  is  it  to  focus  attention  to  the  job  at  hand  while  wearing 
IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

36.  How  difficult  is  it  to  quickly  perform  logical  processes  while  wearing 
IPE  (e.g.,  math,  technical  procedures)? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

37.  Overall ,  how  difficult  is  it  to  remember  things  while  wearing  IPE? 

1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 

38.  How  difficult  is  it  to  understand  what  someone  is  saying  while  wearing 
IPE? 

1>  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 
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43.  Using  the  following  scale,  please  determine  how  difficult  it  is  to 
perform  each  task  while  wearing  IPE: 


« 


1.  Not  difficult. 

2.  Somewhat  difficult. 

3.  Difficult. 

4.  Very  difficult. 

5.  Most  difficult. 


Making  Keystrokes  (typewriter,  computer,  calculator) 

Pushing  Buttons,  knobs 

Flicking  Switches 

Using  Clamps 

Manipulating  Levers 

Using  Screwdrivers,  tools 

Turning  Dials 

Using  Keys 

Using  a  Firm  Grip 

Making  Steady  Wrist  Movements 

Aligning  Equipment 

Tracking  Objects  Across  a  Screen 

Inserting  Objects  or  Instruments  into  Small  Places 
Threading  Objects  (Needles,  Stripped  Bolts) 
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Timebased  Analysis  of  Significant  Coordinated  Operations 

(TASCO) 

by  J.  Armstrong 
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OPERATIO 


1.0  INTRODUCTION 


A  systems  approach  to  critical  task  analysis  (SACPATA )  was 
developed  during  1985  through  applications  of  human  factors 
engineering  and  avionics  technical  skills  in  the  Integrated  Faci¬ 
lity  for  Avionics  Systems  Test  ( IFAST )  facility  located  at  Edwards 
Air  Force  Base.  SACPATA  was  used  to  support  the  F-16  MSIP  and 
AFTI/F-1 6  pilot-avionic  system  task  integration  analyses. 

SACPATA  provided  a  systematic  and  " forward-lookinq”  process 
suitable  for  identifying  and  modifying  pilot-avionic  task  interface. 
The  SACPATA  approach  calls  for  a  pilot-avionic  task  analysis  to 
begin  in  a  conceptual  design  of  the  avionic  system  and  continue 
through  development  test  and  operational  usage. 

Normally,  a  critical  task  analysis  was  accomplished  piecemeal 
using  only  "after-the-fact"  information.  Over  the  years  this 
method  worked  well  because  the  need  for  critical  task  analysis  was 
primarily  for  training  purposes.  During  those  years,  most  fliqht 
test  programs  were  characterized  by  a  "fly-fix-fly"  philosophy. 
The  changes  and  modifications  that  resulted  had  little  effect  on 
critical  task  analyses.  However,  with  complex  integrated  avionic 
systems  featuring  computerized  master  modes  and  submodes,  multi¬ 
function/head-up  displays  with  many  formats  and  menu  options,  it 
is  not  "cost-effective"  to  wait  until  full-scale  development  flight 
tests  to  initiate  an  "after-the-fact"  critical  pilot-avionic  task 
analyses . 

In  addition,  new  avionic  system  designs  have  become  more  com¬ 
plicated  and  problems  and  difficulties  can  be  encountered  durinq 
flight  tests  changing  master  modes  and  submodes,  setting  up  display 
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formats,  and  menu  paging  to  prepare  weapons  for  launch.  It  is 
often  a  real  challenge  to  setup  sensor  modes  and  cockpit  displays 
in  the  time  required  to  accomplish  the  mission  task  flow. 

It  was  apparent  that  something  had  to  be  done  to  resolve 
critical  pilot-avionic  task  difficulty  before  flight  testing.  A 
systems  approach  (SACPATA)  was  drafted  that  outlined  the  techniques 
which  could  be  used  during  avionic  systems  development  to  identify 
pilot-avionic  task  choke  points  and  problem  areas. 

SACPATA  approach  called  for  initiating  the  critical  pilot- 
avionic  task  analysis  in  design  concept  and  extending  it  through  de¬ 
velopmental  stages.  SACPATA  was  characterized  by  the  "identify- 
analyze-modif y"  method  of  pilot-avionic  task  interfacing.  The 
SACPATA  goal  during  early  design  was  to  eliminate  or  reduce  pilot- 
avionic  task  problems  "before-the-fact"  to  prevent  them  from  being 
designed  into  the  system. 

In  view  of  the  fact  that  most  pilot-avionic  system  task  over¬ 
loads  have  been  due  to  avionics  system  design,  the  main  focus  of 
SACPATA  was  placed  on  identifying  pilot-avionic  system  interface 
problems  during  development.  The  pilot-task  overloads  must  be 
eliminated  or  reduced  to  an  "acceptable"  level  prior  to  flight  test 
and  evaluation. 

SACPATA  should  continue  through  the  development  and  operational 
phases.  The  core  of  the  systems  approach  includes  tracking  critical 
pilot-avionic  tasks  to  assure  that  task  problems  (hazards)  do  not 
"unacceptably"  affect  mission  accomplishment. 

2.0  AVIONIC  SYSTEM  CONCEPT  AND  DEFINITION  PHASES 

During  system  concept  ^nd  definition  phases,  pilot-avionic 
critical  task  identification  and  preliminary  avionic  system  design 
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should  be  accomplished  simultaneously.  Pilot-avionic  task  decision 
trees  and  mission  task  flow  simulation  models  can  provide  a  com¬ 
prehensive  investigation  of  pilot-task  choke  points  and  pilot- 
avionics  integration  problem  areas  (hazards)  that  are  associated 
with  the  preliminary  design.  The  results  of  SACPATA  should  be 
reported  in  detail  during  the  concept  and  definition  phase  design 
reviews . 

To  produce  a  "problem  free"  task  design  that  meets  the  criteria 
derived  in  the  concept  phase,  a  clear  definition  of  mission  critical 
pilot-avionic  task  problems  must  be  made.  The  definition  must  in¬ 
clude  "cause  and  effect"  relationship,  and  the  likelihood  and  seve¬ 
rity  of  the  pilot-avionic  task  problem.  If  a  pilot-avionic  task 
problem  persists,  alternative  system  designs  must  be  considered  as 
the  first  means  to  eliminate  the  problem. 

To  obtain  a  clear  definition  of  critical  pilot-avionic  task 
problems,  the  mission  task  flow  analysis  must  be  accomplished.  The 
process  outlined  in  the  Timebased  Analysis  of  Significant  Coord¬ 
inated  Operations  (TASCO),  "A  Cockpit  Workload  Analysis  Technique" 
document  can  be  used  to  initially  identify  the  pilot-avionic  task 
overload  and  potential  mission  critical  task  hazards.  The  timely 
evaluation  of  potential  mission  critical  task  hazards  was  a  key 
element  in  the  overall  human  factors  and  avionics  system  integration 
analysis . 

2.1  AVIONIC  SYSTEM  DEVELOPMENT  PHASE 


The  development  phase  should  include  evaluation  of  pilot-task 
interfaces  through  prototype  analysis  and  preliminary  design  testing. 
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Since  the  design  at  this  time  is  more  complete,  detailed  pilot- 
avionic  system  information  can  be  placed  into  the  critical  pilot- 
avionic  task  analyses. 

Precise  pilot-avionic  system  evaluations  are  required  to  assess 
the  need  and  the  acceptability  of  the  manual  master  mode  and  display 
format  configuration  change  and  the  automatic  cockpit  configuration. 

Corrective  action  must  be  taken  to  relieve  pilot-task  overloads 
in  systems  design  and  the  pilot-task  workload  level  "acceptabili¬ 
ty"  must  be  verified.  Close  coordination  with  avionic  disciplines 
should  be  exercised  and  specific  actions  initiated  to  eliminate  or 
reduce  harmful  effects  of  pilot-avionics  task  overload  and/or  task 
hazards  on  the  mission. 

Completion  of  the  development  phase  should  lead  to  a  "GO/NO- 
GO"  decision  of  the  final  system  design  before  the  actual  production 
begins.  The  analyst's  ability  to  make  the  correct  GO/NO-GO  decision 
will  be  based  upon  completion  of  the  preliminary  pilot-avionic  task 
analysis  and  preliminary  test  results. 

2.2  AVIONIC  SYSTEM  PRODUCTION  AND  DEPLOYMENT  PHASES 

An  updating  of  potential  task  hazards  must  be  continued  throuqh 
the  full-scale  evaluation  and  test  phase.  The  comprehensive  review 
of  the  preliminary  pilot-avionic  task  analysis  must  be  accomplished 
to  verify  that  appropriate  corrective  actions  have  been  taken  to 
eliminate  or  reduce  the  effects  of  the  task  hazard.  Modifications 
and  changes  in  system  design  to  eliminate  or  reduce  potential  pilot- 
avionic  task  hazards  will  be  subject  to  review  and  verification 
during  final  design  acceptance  meetings. 
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3.0  SAC PAT A  SEQUENCE  OF  PILOT-AVIONIC  TASK  HAZARD  REMOVAL  AND  CON¬ 
TROL 

SACPATA  will  identify  and  document  pilot-avionic  task  work  load 
levels  in  the  design  under  development.  SACPATA  data,  resulting 
from  design  evolution  and  evaluation,  can  play  an  important  role  in 
system  acceptance. 

SACPATA  information  must  be  carried  into  production  and  deploy¬ 
ment  phases  to  follow-up  on  potential  task  hazards  that  may  remain 
in  the  system's  design  or  be  uncovered  through  operational  usage. 

The  sequence  of  pilot-avionic  task  hazard  removal  and  control 
should  follow  a  system  of  prioritized  activities.  The  most  preferred 
activity  sequence  is  as  follows: 

a.  Design  to  eliminate  pilot-avionic  task  hazards 

b.  Design  to  reduce  critical  task  hazards  to  an  acceptable 
level 

c.  Control  critical  task  hazards  through  special  task  proce¬ 
dures 

d.  Control  critical  task  hazards  through  special  task  training. 

Design  to  eliminate  task  hazards  is  an  ambitious  qoal,  even 

though  some  seqments  of  the  mission  require  strict  "design-out"  of 
all  pilot-avionic  task  hazards  to  avoid  an  "unacceptable"  loss. 

Design  to  reduce  task  hazards  to  an  "acceptable"  level  implies 
a  residual  problem  left  in  the  desiqn  of  the  system.  The  potential 
of  the  task  problem  may  be  reduced  in  magnitude  by  reduction  of  its 
probability  and  severity.  Also,  an  "acceptable"  work  load  level 
in  one  mission  segment  may  be  totally  "unacceptable"  in  another  mis¬ 
sion  segment.  Task  hazard  "acceptability"  must  be  weighed  in  terms 
of  risk  benefit  factors  for  each  segment  of  the  mission. 


Special  procedures  and  training  can  be  used  to  control  a  task 
problem  if  design  methods  should  fail.  In  this  case,  special 
procedures  must  be  clearly  detailed  and  pilots  specially  trained. 
This  is  not  a  preferred  means  of  reducing  task  hazards,  but  may  be 
the  only  avenue  remaining.  Design  changes  or  modifications  after 
definition  phase  may  be  complex  or  costly. 

4.0  AVIONIC  SYSTEM  LIFE-CYCLE 

A  system's  life-cycle  is  normally  separated  into  distinguishable 
phases.  The  most  common  phases  are  the  concept,  definition,  develop¬ 
ment,  production,  and  deployment  phases.  During  these  five  phases, 
SACPATA  effort  should  address  the  following  human  factors  engineering 
and  avionics  system  integration  aspects  as  shown  in  Table  1. 

Table  1 

SYSTEMS  APPROACH  TO  CRITICAL  PILOT-AVIONICS  TASK  ANALYSIS 

(SACPATA) 

HUMAN  FACTORS  ENGINEERING  AND  AVIONICS  SYSTEM  INTEGRATION 


Phase 

Aspects 

Concept 

Pilot  and  Avionic  System  Interface  Conceptual  Re¬ 
search 

Definition 

Critical  Pilot-Avionic  Task/System  Design  Verifica¬ 
tion 

Development 

Mission  Critical  Pilot  Tasking/Full-Scale  Develop¬ 
ment  Verification 

Production 

Mission  Critical  Pilot.  Tasking/Operational  Valida¬ 
tion 

Deployment 

Mission  Critical  Task/Mission  Objective  Validation 

A  relative  amount  of  pilot-avionic  task  analysis  (SACPATA) 
effort  that  should  be  expected  and  planned  during  each  life-cycle 
phase  is  shown  in  Figure  1. 
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Figure  1  SACPATA  Effort  During  Avionic  System  Life  Cycle 


5.0  SACPATA  PRIORITY  DIAGRAM  FOR  TASK  HAZARD  ELIMINATION  AND/OR 
REDUCTION 


The  priorities  that  should  be  used  to  eliminate  or  reduce  pilot- 


avionic  task  hazard  by  design  is  shown  in  Figure  2.  If  this  fails. 


control  task  problems  through  special  procedures  or  special  training. 


5.1  SACPATA  OUTPUTS  AND  SUMMARY 


The  outputs  from  a  properly  accomplished  SACPATA  can  form  a 


basis  for  specifying  automatic  configuration  changes  over  manual 


control.  The  data  can  provide  operational  and  work-around  pro¬ 


cedures,  as  well  as,  risk-benefit  factors  for  a  meaningful  human 


engineering  and  avionics  integration  analyses. 


Expected  outputs  from  critical  pilot-avionic  task  analyses  are 


shown  in  Figure  3. 


5.2  CRITICAL  PILOT-AVIONIC  TASK  ANALYSIS  SUMMARY 


The  task  analysis  should  not  be  regarded  as  a  task  "failure" 


analysis.  That  would  be  an  "after-the-fact"  approach.  Critical 
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Figure  2  Pilot-Avionic  Task  Hazard  Elimination  and/or  Reduction 
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Figure  3  Critical  Task  Analyses  (SACPATA)  Outputs 


task  analysis  should  occur  "before-the-fact"  and  be  regarded  as 
critical  task  hazard  analysis.  The  analytical  methods  used  are 
specialized  analysis  techniques  involving  probability  principles, 
risk-benefit  disciplines,  and  the  logic  theory. 

Pilot-avionic  task  procedures  must  be  designed  and  built-in  to 
an  aircraft  just  as  performance,  stability,  structural  integrity, 
safety,  etc.  A  human  factors  engineering  and  avionics  integration 
group  should  be  an  important  part  of  an  aircraft  manufacturer's 
organization,  just  as  stress,  aerodynamics,  weight  and  balance,  and 
armament  groups. 

Critical  task  analyses  are  not  unique,  but  they  have  not 
flourished  or  been  formally  adopted  as  an  aid  in  the  design  of  man- 
machine  systems.  In  fact,  critical  task  analyses  concepts  have 
not  been  clearly  defined  by  Government  agencies  contracting  in  the 
civilian  aerospace  field.  SACPATA  represents  a  first  step  in 
establishing  critical  task  analyses  definition. 
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5.3  COMPUTER-AIDED  CRITICAL  PILOT-AVIONIC  TASK  ANALYSES 

Computer-aided  (or  computer-based)  evaluation  and  testing  in 
critical  task  analyses  can  form  vital  links  between  the  system 
preliminary  design,  objective  task  flows,  and  cockpit  work  load 
assessment  logic  structures.  A  mission  task  flow  evaluation  and 
testing,  that  was  properly  conducted,  would  provide  the  pilot's  in¬ 
puts  to  the  system  model  and  also  serve  as  an  entry  point  for  operator 
procedures  into  the  task  analysis.  Figure  4  illustrates  a  mission 
critical  task  flow  evaluation  and  testing  process  in  broad-scale 
overview. 


Figure  4  Computer-Aided  Mission  Critical 
Task  Flow  Evaluation/Testing 


The  mission  critical  task  flow  evaluation  and  testing  examines 
the  mission  capabilities  and  objectives  with  respect  to  pilot-avi¬ 
onic  system  capabilities.  Aircraft  and  mission  capabilities  can  be 
identified  early  in  the  concept  phase.  Preliminary  man-machine  task 
allocation  can  be  developed  to  support  the  aircraft  and  mission 
capabilities.  The  risk-benefit  analysis  is  used  in  a  preliminary 


10 


Critical  Task  Analyses  (SACPATA)  Outputs 


MISSION 

OBJECTIVE 

DATABASE 

COCKPIT 

WORKLOAD 

ANALYSIS 

MODEL 

COMPUTER 

AIDED 

EVALUATION 
&  TESTING 

& 

TASK  FLOW 
DATA  FILE 
STRUCTURE 

MISSION 

CRITICAL 

TASK 

ASSIGNMENTS 

f 

235 


I 

» 

form  to  reach  conclusions  about  the  importance  of  the  tasking 
assignments.  Three  basic  questions  should  be  answered  at  the  close 
of  the  concept  phase. 

a.  Have  the  task  hazards  associated  with  system  desiqn  been 
discovered  and  evaluated  to  assure  pilot-aircraft-mission 
objectives  have  been  met? 

b.  Have  risk-benefit  analyses  been  initiated  to  establish  the 
means  of-  task  hazard  assessment  and  control? 

c.  Are  critical  pilot-avionic  task  design  requirements  estab¬ 
lished  for  the  concept  so  that  the  next  phase  of  system 
definition  can  be  initiated? 

6.0  PILOT-AVIONIC  SYSTEM  CAPABILITIES  AND  MISSION  OBJECTIVES 

When  the  avionic  system  is  evaluated  in  the  light  of  mission 
objectives,  strengths  and  weaknesses  surface  as  pilot-task  decision 
trees  are  constructed.  Pilot  opinions  and  recommendations  are  inter- 
played  with  mission  objectives  durinq  the  evaluation  of  avionic 
system  capabilities.  The  SACPATA  approach  calls  for  the  examination 
of  the  total  pilot-aircraft-mission  system  during  the  early  stages 
of  avionic  system  development.  The  interplay  of  pilot,  aircraft, 
and  mission  is  shown  in  Figure  5. 


Figure  5  Critical  Task  Analysis  Factors 
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In  addition  to  mission  critical  pilot-avionic  task  identifica¬ 
tion,  operator  evaluation  and  testing  provides  many  salient  products 
useful  in  the  task  analysis  which  include  the  following: 

a.  Strengths  and  weaknesses  of  the  avionics  system  surface 
during  the  critical  pilot-avionic  task  examination. 

b.  Alternative  techniques  and  work-around  procedures  evolve 
as  the  pilot-avionic  task  decision  trees  are  developed. 

c.  Seldom  utilized  cockpit  procedures  and  display  modes  are 
keynoted . 

d.  Pass  or  fail  criteria  for  operator  task  performance  are  es¬ 
tablished  . 

e.  Operator  task  overloads  and  mental  stress  points  are 
highlighted. 

7.0  SACPATA  COMPUTER-AIDED  CRITICAL  PILOT-AVIONIC  TASK  ANALYSES 
MODEL 

The  mission  scenario  is  evaluated  to  produce  a  "benchmark 
sortie"  having  operational  characteristics.  The  benchmark  sortie 
analysis  provides  cockpit  mission  critical  task  assignments  through 
tasking  criteria  application.  The  pilot's  inputs,  through  computer- 
aided  evaluation  and  testing,  are  examined  during  cockpit  task  an¬ 
alysis  to  determine  risk-benefit  probabilities  for  the  most  criti¬ 
cal  mission  flow  pilot-avionic  tasks. 

Figure  6  shows  where  and  how  computer-aided  mission  critical 
task  flow  evaluation  and  testing  fits  into  the  broad-scale  pilot- 
avionic  task  analysis  model. 

7.1  CRITICAL  PILOT-AVIONIC  TASK  HAZARD  PROBABILITY  ANALYSES 

Critical  task  pilot-avionic  task  analyses  can  assess  the  degree 
of  task  "hazard"  in  a  system  either  qualitatively  or  quantitatively. 

12 
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Figure  6  Computer-Aided  Critical  Pilot-Avionic  Task  Analyses  Model 


A  qualitative  assessment  uses  the  judgemental  approach  which  may  lack 
sophistication  and  allow  biased  data  to  enter  the  analysis.  A  quan¬ 
titative  approach  uses  computerized  algorithms  requiring  the  appli¬ 
cation  of  rigorous  logic  for  cause-effect  relations  and  probability 
predictions . 

The  critical  pilot-avionic  task  "hazard"  assessment  should  be 
initiated  "before-the-fact"  and  certain  statistical  methods  can  be 
used  to  account  for  any  uncertainty  by  probability  predictions. 
There  are  two  basic  types  of  probability  statistics.  One  form,  called 
"a  posteriori"  probability,  is  developed  by  conducting  a  test  or  an 
experiment  and  observing  the  outcome.  Sample  data  that  are  collected 
during  the  test  are  used  to  derive  the  probability.  The  data  may  be 
collected  from  "real-world"  events.  The  following  sample  aircraft 
accident  data  are  used  for  "a  posteriori"  probability  prediction. 


Aircraft  Accident  Sample  Data 
5  -  Resulted  in  death 
10  -  Resulted  in  severe  injury 
30  -  Resulted  in  moderate  injury 
75  -  Resulted  in  minor  injury 
250  -  Resulted  in  no  injuries 
370  Total  number  of  accidents 

Probability  of  Death  (d),  given  that  one  is  in  an  aircraft  ac¬ 
cident,  is: 

P(d )  =  5/370  *  0.0135 

An  "a  priori"  probability  can  be  specified  by  the  evident  na¬ 
ture  of  the  events  from  which  they  emerge.  The  "a  priori"  probability 
of  a  head  (h)  coming  up  from  a  toss  of  a  fair  coin  can  be  calculated 
as  0.5. 

P(h)  +  P(t)  =  0.5  +  0.5  =  1.0 
P(h)  =  0.5 

The  critical  pilot-avionic  task  analysis  uses  "a  priori" 
probabilities  for  making  decisions  concerning  the  acceptability  of 
pilot-task  "hazard." 

7.2  MISSION  CRITICAL  PILOT-AVIONIC  TASK  HAZARD 

A  "hazard"  may  be  simply  defined  as  "a  potential  for  doing 
harm."  The  "harm"  in  a  pilot-avionic  task  is  not  being  in  the 
correct  sensor  mode  for  taking  a  navigation  fix  which  results  in  a 
missed  way  point,  or  not  being  in  the  right  weapon  mode  for  launching 
a  missile  and  missing  a  target.  The  word  "potential"  is  important 
to  the  definition  of  the  critical  task  hazard  as  a  task  analyst  must 
control  them  to  an  "acceptable"  level  by  using  one  of  several  means. 


Task  hazards  possess  two  inherent  properties;  likelihood  and 
severity.  If  either  of  these  two  properties  has  no  value,  then  the 
task  hazard  has  no  potential  for  harm  and  is  intrinsically  safe. 
Whenever  a  critical  task  hazard  is  found  through  the  task  hazard 
analysis  process  that  is  not  intrinsically  safe,  it  must  be  eval¬ 
uated  for  acceptability  in  an  operational  mission  environment.  If 
the  task  hazard  is  "unacceptable,"  it  must  be  dealt  with  by  system 
design  changes,  mod if icat ions ,  or  controlled  through  countermea¬ 
sures  such  as  work-around  procedures. 

A  system  of  critical  task  hazard  severity  can  be  setup  similar 
in  form  to  the  four  main  categories  used  in  Table  2.  These  catego¬ 
ries  are  used  in  the  aerospace  industry  to  deal  with  safety  hazards. 


Table  2 

SAFETY  HAZARD  CLASSIFICATION 


Cateqory 

Name 

Characteristics 

I 

Catastrophic 

Death  -  loss  of  system 

II 

Critical 

Severe  injury  -  major  damage 

III 

Marginal 

Minor  injury  -  minor  damage 

IV 

Negligible 

No  injury  -  no  damage 

As  mentioned  previously,  critical  task  hazard  probability  refers 
to  the  probability  occurrence  of  an  event  (or  the  non-occurrence  of 
an  event),  that  results  in  harm.  In  the  case  of  mission  task  flow, 
the  "harm"  may  be  due  to  being  out-of-mode  or  not  in  the  appropriate 
display  format  to  complete  the  mission  task. 

Early  in  the  system  design  stage,  there  may  be  insufficient 
information  to  perform  precise  critical  pilot-avionic  task  analysis. 
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However,  as  the  system  matures  and  information  is  available, 
probability  evaluation  can  be  accomplished  with  more  precision. 
7.3  CRITICAL  TASK  HAZARD  UNDESIRABILITY  AND  SEVERITY  CLASSIFICATION 
A  critical  task  hazard  measure  of  undesirability  (or  disutility) 
can  be  related  to  its  level  of  severity  and  probability.  Table  3 
illustrates  relative  degree  of  disutility  of  task  hazards  as  a 
function  of  probability  and  severity. 

Table  3 

CRITICAL  TASK  HAZARD  (DISUTILITY)  UNDESIRABILITY 


Probability 

Severity 

LOW 

HIGH 

LOW 

Low-Low 
{ low) 

Low-High 
(moderate ) 

HIGH 

High-Low 
(moderate ) 

High-High 

(high) 

The  least  "undesirable"  task  hazard  is  in  the  upper  left  corner 
and  has  a  low  value  (Low-Low)  of  disutility.  The  lower  right  corner 
has  the  highest  relative  value  of  disutility  (High-High)  and  is  the 
most  "undesirable"  task  hazard. 

The  critical  pilot-avionic  task  "hazards"  are  similar  to  safety 
hazards.  A  severity  classification  can  be  applied  to  pilot-avionic 
task  hazards,  but  the  categories  must  be  expanded  for  a  broader- 
range  "harm  potential"  to  the  mission.  Table  4  shows  a  "broader- 
scale"  severity  classification  used  for  the  critical  pilot-avionic 
task  hazard  severity  classification. 
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SAFETY  HAZARD  CLASSIFICATION 


Cateaorv 


I 

II 

III 

IV 


Name 

Characteristics 

Catastrophic 

Critical 

Marginal 

Negligible 

Death  -  loss  of  system 

Severe  injury  -  major  damage 

Minor  injury  -  minor  damage 

No  injury  -  no  damage 

CRITICAL  TASK  HAZARD  (DISUTILITY)  UNDESIRABILITY 


Probability 


Severit 


HIGH 


HIGH 


Low-Low 
( low) 


High-Low 
(moderate ) 


Low-High 
(moderate ) 


High-High 

(high) 


Table  4 


CRITICAL  PILOT-AVIONIC  TASK  HAZARD  SEVERITY  CLASSIFICATION 


7.4  CRITICAL  PILOT-AVIONIC  TASK  HAZARD  QUALITATIVE  PROBABILITY 
RANKING 

The  qualitative  critical  task  hazard  probability  ranking  pro¬ 
vides  rating  scales  for  occurrence  of  task  hazards  and  is  used  in 
risk-benefit  definition  to  determine  those  areas  where  task  modifi¬ 
cation  must  be  accomplished.  In  some  areas,  automatic  change  of 
master  mode  and  submode,  and  display  configuration  priority  have  been 
used  to  reduce  or  control  "unacceptable"  task  hazards.  In  cases  of 
automatic  mode  changes  (pre-assigned  display  formats  and  sensor 
modes)  an  evaluation  must  be  made  to  determine  their  actual  impact 
on  the  pilot's  mission  task  flow.  Increases  in  automaticity  can 
reduce  the  pilot's  flexibility,  do  harm  to  his  mission  capability, 
or  result  in  an  unacceptable  system  configuration  in  another 
segment  of  the  mission. 

Table  5  provides  a  critical  pilot-avionic  task  hazard  qualita¬ 
tive  probability  ranking  that  can  be  used  in  the  SACPATA. 
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Table  5 

CRITICAL  PILOT-AVIONIC  TASK  HAZARD  QUALITATIVE  PROBABILITY  RANKING 


Occurrence 

Probability 

Test 

Experience 

Operational 

Usage 

Frequently 

probable 

6 

Likely  to  occur 
frequently 

Likely  to  be  ex¬ 
perienced  con¬ 
tinuously 

Reasonably 

probable 

5 

Will  occur  sev¬ 
eral  times 

Will  occur  many 
times 

Occas ionally 
probable 

4 

Likely  to  occur 
several  times 

Will  occur  sev¬ 
eral  times 

Remotely 

probable 

3 

So  unlikely,  its 
not  expected 

Unlikely  to  oc¬ 
cur,  but  possible 

Extremely 

improbable 

2 

Occurrence  times 
next  to  zero 

Occurrence  only 
after  other  com¬ 
plications 

Impossible 
to  occur 

1 

Physically  impos¬ 
sible 

Physically  im¬ 

possible 

7.5  CRITICAL  PILOT-AVIONIC  TASK  HAZARD  INDEX 

A  critical  pilot-avionic  task  hazard  index  can  be  developed 
from  the  likelihood  (probability)  and  the  undesirable  (severity) 
effects.  This  task  hazard  index  can  then  be  used  to  differentiate 
the  effect  of  the  task  hazard  on  the  mission. 

Figure  7  uses  the  probability  (likelihood)  and  severity  (unde¬ 
sirable  effects)  indices  to  develop  "critical"  pilot-avionic  task 
hazard  indices  for  differentiating  harmful  effects  on  the  mission. 

The  "critical"  pilot-avionic  task  hazard  index  is  the  product 
of  probability  and  severity  points.  Having  formed  the  basic  matrix 
of  critical  task  hazard  indices,  the  analyst  can  designate  the  level 
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CRITICAL  PILOT-AVIONIC  TASK  HAZARD  SEVERITY  CLASSIFICATION 


Severity  Index 

Effect  on  Mission 

. 

6 

No  effect  on  mission  or  aircraft 

5 

Minor  ef f ect-(missed  navigation  waypoint) 

4 

Major  ef f ect-(missed  target  initial  point) 

3 

Major  ef f ect-(missed  target  of  opportunity) 

2 

Major  effect- (missed  mission  objective  target) 

1 

Major  effect-( aircraft  damage  or  loss) 

CRITICAL  PILOT-AVIONIC  TASK  HAZARD  QUALITATIVE  PROBABILITY 
RANKING 


CRITICAL  PILOT-AVIONIC  TASK  HAZARD  QUALITATIVE  PROBABILITY  RANKING 


Occurrence 

Probability 

Test 

Experience 

Operational 

Usage 

Frequently 

probable 

6 

Likely  to  occur 
frequently 

Likely  to  be  ex¬ 
perienced  con¬ 
tinuously 

Reasonably 

5 

Will  occur  sev- 

Will  occur  many 

probable 

eral  times 

times 

Occas ionally 
probable 

4 

Likely  to  occur 
several  times 

Will  occur  sev¬ 
eral  times 

Remotely 

probable 

3 

So  unlikely,  its 
not  expected 

Unlikely  to  oc¬ 
cur,  but  possible 

Extremely 

improbable 

2 

Occurrence  times 
next  to  zero 

Occurrence  only 
after  other  com¬ 
plications 

Impossible 
to  occur 

1 

Physically  impos¬ 
sible 

Physically  im¬ 

possible 

CRITICAL  PILOT-AVIONIC  TASK  HAZARD  INDEX 
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PROBABILITY 

SEVERITY  INDICES 

INDICES 

1 

2 

3 

4 

5 

6 

1 
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Figure  7  Critical  Pilot-Avionic  Task  Hazard  Indices 


of  task  hazard  index  that  is  "unacceptable"  or  "critical"  for  that 
segment  of  the  mission  (e.g.,10),  as  shown  by  the  dotted  line  in 
Figure  7. 

8.0  CRITICAL  PI LOT- AVIONIC  TASK  DECISION  TREE  ANALYSIS  METHOD 

The  decision  tree  is  fundamentally  a  Boolean  logic  model  that 
depicts  the  relationship  between  events  in  a  system  that  lead  to  a 
final  outcome  event.  The  final  outcome  event  of  the  pilot-avionic 
task  tree  intersects  a  mission  task  flow  at  the  head  (top-level) 
event.  Subelement  events  are  below  the  head  event  and  form  logical 
occurrences  required  to  achieve  final  outcome  of  the  head  event. 

The  pilot-avionic  task  decision  tree  and  mission  task  flow 
intersection,  as  used  in  SACPATA,  is  depicted  in  Figure  8. 

While  the  method  carries  in  the  title  "Decision  Tree,"  con¬ 
noting  a  Top-Level  Head  Event  as  a  decision,  the  tree  may  be 
constructed  with  the  top-level  event  such  as  the  one  depicted 
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in  Figure  8.  In  that  case,  the  top-level  event  was  "Verify  AA/LCOS 
Mode/Format . "  The  tree  would  then  become  the  pilot-avionic  system 
interactions  required  to  produce  or  satisfy  the  top-level  (head) 
event . 


PERFORM 

NAVIGATION 

UPDATE 


CONFIGURE 

FOR 

AA  COMBAT 


VERIFY 

AA/LCOS 

MODE/FORMAT 


PERFORM 


COMBAT 


(DECISION  TREE) 
(HEAD  EVENT) 


Figure  8  Mission  Task  Flow  and  Critical 

Pilot-Avionic  Task  Intersection 


8.1  PILOT-AVIONIC  TASK  DECISION  TREE  ( SAC PAT A )  APPROACH 

The  task  decision  tree  approach  is  a  systematic,  descriptive 
form  of  analysis  that  may  be  applied  to  task  analyses.  It  can  be 
useful  in  the  early  design  phase  of  a  new  system  and  particularly 
important  for  analyzing  operational  systems.  The  decision  tree 
method  allows  the  analyst  to  evaluate  alternatives  and  judge  accept¬ 
able  trade-offs  among  them.  The  decision  tree  method  has  the  power 
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of  deduction  where  combination  subelement  events  should  be  considered 
in  the  causal  chain.  The  interactions  between  events  and  subsystems 
form  vital  parts  for  understanding  how  the  system  functions,  or  how 
it  should  function. 

As  mentioned  previously,  the  analysis  of  a  system  through  a 
decision  tree  can  result  in  either  a  qualitative  or  a  quantitative 
output,  or  both  whichever  is  most  advantageous  for  the  analyst.  The 
qualitative  output  results  in  sets  of  events  that  effect  the  top- 
level  event  and  ranking  of  each  event  for  sensitivity  of  the  top- 
level  event  to  its  occurrence  or  importance. 

The  analyst  may  obtain  quantitative  probabilities  or  rates  of 
occurrence  of  the  basic  events  to  obtain  a  more  precise  ranking  of 
the  importance  of  each  event,  as  well  as  other  probalistic  measures. 

The  pilot-avionic  task  decision  tree  is  prepared  by  a  compu¬ 
terized  function  that  diagrams  contributing  factors  linked  through 
logic  gates  to  a  head  event.  The  task  decision  tree  clearly, 
precisely,  and  concisely  defines  the  top-level  mission  flow  head 
event  for  which  nonoccurrence  would  be  critical  to  the  mission  task 
flow. 

Each  subelement  of  the  event  that  is  capable  of  producing  an 
occurrence  or  adding  to  the  capability  of  producing  the  occurrence 
of  the  event  is  examined.  Then  a  determination  is  made  as  to  how 
its  failure  to  occur  could  contribute  to  the  failure  of  the  head 
event  to  occur.  When  more  than  one  subelement  on  a  task  tree  could 
contribute  to  the  same  effect  on  the  event,  a  determination  is  made 
as  to  whether  input  subelements  must  act  in  combination  (-AND- 
relationship )  to  produce  the  effect,  or  whether  they  may  act  singly 
(-OR-  relationship)  to  produce  the  same  effect.  The  relationships 


for  the  pilot-avionic  task  decision  tree  are  simplified,  as  much  as 
possible,  to  ease  construction. 

Mathematical  expressions  for  representing  pilot-avionic  task 
decision  trees  are  developed  using  simple  Boolean  algebra.  Pos¬ 
sibility  of  each  subdlement  via  a  logic  model  producing  nonoccur¬ 
rence  of  a  head  event  is  determined  by  probability  density  and  dis¬ 
tribution  functions. 

As  data  becomes  available  from  simulations  and  flight  tests 
designed  to  examine  the  mission  flow  tasking,  the  basic  function 
probabilities  should  be  modified  or  appropriately  adjusted  to 
reduce  risk  for  nonoccurrence  of  the  head  event. 

8.2  PILOT-AVIONIC  TASK  DECISION  TREE  ASSUMPTIONS  AND  STIPULATIONS 

Additional  fallouts  which  may  result  from  using  the  task  deci¬ 
sion  tree  are: 

a.  Identification  of  critical  mode  changes  requiring  automati- 
city . 

b.  Discovery  of  hidden  or  latent  single-point  failure  probabil¬ 
ities  . 

c.  Determination  of  the  most  critical  and  most  probable  sequence 
of  events  that  could  lead  to  an  out-of-mode  condition. 

The  pilot-avionic  task  decision  tree  is  basically  a  logic  dia¬ 
gram  that  shows  various  subelements  that  culminate  in  the  predeter¬ 
mined  head  event  occurrence  or  produce  nonoccurrence.  Certain 
preliminary  assumptions  and  stipulations  should  be  made  concerning 
the  characteristics,  conditions,  and  actions  involving  the  head  event 


t>* 


252 


and  subelements.  These  assumptions  and  stipulations  are  stated  as 
follows : 

a.  Basic  subelement  events  producing  the  same  effect  on  the 
head  event  must  be  independent  of  each  other. 

b.  Head  events  and  subelement  events  have  only  two  conditional 
modes  -  totally  operative  or  "in-mode",  or  totally  non¬ 
operative  or  "out-of-mode."  No  partial  operation  or  par¬ 
tially  "in-mode"  is  included  in  the  analysis. 

c.  Risk-benefit  analyses  must  use  "worse-case"  severity  in¬ 
dices  for  nonoccurrence  of  each  top-level  or  subelement 
event . 

As  mentioned  before,  the  analyst  may  obtain  quantitative  pro¬ 
babilities  or  rates  of  nonoccurrence  of  basic  events.  He  may  also 
obtain  a  more  precise  ranking  of  the  importance  of  each  event,  as 
well  as  other  probabilistic  measures. 

8.3  PILOT-AVIONIC  TASK  DECISION  TREE  SYNTHESIS 

Task  decision  tree  synthesizing  proceeds  fundamentally  by 
the  analyst's  repeated  questions  asking  what  are  the  "real"  deci¬ 
sions  that  have  to  be  made  for  each  of  the  basic  events.  It  is 
obvious  that  the  analyst  must  not  only  have  a  thorough  understand¬ 
ing  of  the  avionic  system,  but  be  well  versed  in  related  areas  of 
knowledge  that  generate  human  factors  engineering  concerns. 

Human  capabilities  and  performance  must  be  carefully  analyzed 
with  system  interfaces.  Intimate  knowledge  of  the  avionic  system 
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requires  long  sessions  with  system  designers  and  operators  to  create 
a  full  understanding  of  design  philosophy  and  methods  of  system 
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operation.  If  subsystems  are  developed  and  provided  by  another 
organization  or  company,  information  may  be  difficult  to  obtain. 
Much  of  the  value  of  the  decision  tree  development  has  to  come  from 
intimate  understandings  as  the  analyst  probes  various  designers  or 
design  teams.  There  may  have  been  little,  or  no  prior  communication 
between  design  teams  or  designers  of  individual  subsystems.  The 
task  analyst  must  bridge  this  gap. 

In  an  attempt  to  simplify  the  design  solutions,  a  system  designer 
usually  will  separate  the  design  into  subdesign  problems.  Such  an 
approach  can  leave  interactive  gaps  for  understanding  full  system 
functions.  The  pilot-avionic  task  decision  tree  structure  with 
enforced  analytical  processes  and  expanded  viewpoint  of  the  analyst 
should  uncover  many  oversights  in  system  design. 

While  computer  codes  are  used  extensively  in  analyzing  the 
pilot-avionic  task  decision  tree  after  it  has  been  developed,  the 
intercourse  between  the  designers  and  analyst  is  the  point  at  which 
the  decision  tree  is  synthesized.  Unfortunately,  there  is  no 
computer-synthesized  task  decision  tree  at  the  present  time. 

In  addition  to  those  advantages  previously  mentioned ,  a  properly 
synthesized  task  decision  tree  can  provide  the  following  advantages: 

a.  Direct  the  analyst  deductively  to  mission  critical  pilot- 
avionic  task  hazard  events. 

b.  Provide  a  graphical  depiction  of  operator  and  avionic  sys¬ 
tem  interface  functions  most  critical  to  mission  accomplish¬ 
ment  . 

c.  Provide  options  for  both  qualitative  and  quantitative 
critical  pilot-avionic  task  analyses. 
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d. 
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Provide  the  analyst  with  an  insight  to  the  avionics  system's 
function  and  behavior. 

The  pilot-avionic  task  decision  tree  synthesis  process  follows 
prescribed  and  finely  detailed  elemental  considerations.  It  forces 
the  analyst  to  further  understand  the  system  beyond  the  level  of 
the  designer  of  any  subsystem  or  component.  Once  the  logic  of  the 
tree  has  been  assembled/  intermediate  steps  for  nonoccurrence  of 
mission  -critical  pilot-avionic  task  events  become  readily  apparent 
to  the  task  analyst. 

8.4  PILOT -AVIONIC  TASK  DECISION  TREE  CONSTRUCTION 

The  construction  of  the  decision  tree  usually  commences  with 
the  top-level  (head)  event  and  proceeds  downward  through  successive 
levels  of  intermediate  effect  steps.  The  analyst  must  determine  at 
each  sublevel  of  events  what  the  next  lower  set  of  events  are  and 
if  they  are  both  sufficient  and  necessary  to  reach  events  at  the 
base  level. 

A  top-level  (head)  event  is  extracted  from  the  mission  task 
flow  analysis.  In  addition  to  determining  the  top-level  event  as 
origin  in  the  development  of  the  pilot-avionic  task  decision  tree, 
the  analyst  must  determine  the  state  of  the  system  at  the  time  it 
is  analyzed  for  occurrence  of  the  top  event. 

For  example,  if  the  analyst  should  choose  "Verify  A-A  Combat 
Configuration"  as  a  top-level  (head)  event,  the  decision  tree  would 
be  different  depending  upon  whether  the  "A-A  Configuration"  would 
mean  gun-firing  or  missile  firing. 
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8.5  PI  LOT -AVIONIC  TASK  DECISION  TREE  SYMBOLOGY 


As  the  analyst  goes  down  the  pilot-avionic  task  decision  tree 
from  event  to  event  determining  "what  the  fundamental  decisions 
are,"  looic  gate  symbols  are  used  to  show  the  logical  relations 
among  events.  The  logic  gate  symbols  have  been  simplified  or 
minimized  for  aiding  construction  of  the  pilot-avionic  task  deci¬ 
sion  tree. 

Logic  gate  symbols  depict  the  logical  relations  among  events 
of  the  tree.  There  are  two  basic  logical  relations  with  several 
subsets  for  each.  The  loqical  -OR-  qate  is  a  logical  relation  that 
requires  occurrence  of  one  of  the  input  events  to  effect  occurrence 
of  the  output  event.  In  most  systems,  if  one  subelement  event 
occurs,  others  are  excluded  from  occurring.  This  form  of  relation¬ 
ship,  through  a  logic  gate  is  referred  to  as  an  exclusive  -OR-. 
The  task  decision  tree  uses  the  exclusive  -OR-  gate  form. 

The  other  principal  logic  gate  is  the  -AND-  gate  which  requires 
that  all  of  the  input  events  must  occur  to  effect  the  output  event 
occurrence.  A  restriction  that  may  be  placed  on  an  -AND-  gate  is  a 
sequence  requirement  that  requires  the  input  events  to  occur  in  an 
order  to  effect  the  output  occurrence.  This  is  called  a  priority 
modification  of  the  -AND-  gate. 

Top  level  and  subelemental  events  are  shown  on  the  tree  as 
rectangles.  If  the  analysis  is  to  be  quantitative,  a  basic  event 
will  be  assigned  a  rating  or  a  probability,  and  the  top-level  event 
will  be  determined  by  Boolean  reduction  of  the  tree. 
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8.6  PILOT-AVIONIC  TASK  DECISION  TREE  ANALYTICAL  PROCESS 

Once  logically  constructed,  the  task  decision  tree  depicts  a 
Boolean  model  of  the  system.  Having  constructed  the  system  model, 
it  must  be  exercised  or  analyzed  if  useful  information  is  to  be 
gained.  The  visual  examination  of  the  pictorial  diagramatic  descrip¬ 
tion  of  the  task  decision  tree  may  be  of  some  use  for  identifying 
task  choke  points.  However,  substantial  information  is  lost  by  just 
a  cursory  examination  of  the  critical  task  processes.  Information 
that  is  available,  only  from  full  analyses,  points  the  way  to  a 
detailed  statement  of  the  system's  present  state  and  recommended 
modifications  for  improvement. 

8.7  BOOLEAN  EQUIVALENT  TREE 


Having  developed  the  Boolean  relationships  for  two  fundamental 
logic  gates  ( -AND-/-OR- ) ,  a  Boolean  equivalent  expression  can  be  de¬ 
veloped  for  each  of  the  task  decision  trees.  If  the  basic  events 
are  independent  and  all  of  the  basic  events  must  occur  to  allow  the 
top-level  event  to  occur,  a  multiplication  rule  applies  as  shown  in 
Figure  9. 


H 


-AND-  P(H)  -  P{A)  P(B) 


A  B 


Figure  9  -and-  Gate 

If  the  basic  events  are  mutually  exclusive,  then  the  rule  of 
addition  applies  as  shown  in  Figure  10. 
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A  B 


Figure  10  -or-  Gate 

8.8  PILOT-AVIONIC  TASK  DECISION  TREE  ABSOLUTE  LOGIC  TRUTH  TABLE 
A  truth  table  can  be  developed  for  the  -OR-  and  -AND-  gates 
which  is  used  to  define  the  input  and  output  words.  Figures  11  and 
12  show  the  truth  tables  for  the  logical  gates  used  in  the  critical 
task  decision  tree. 


(C)  TRUTH  TABLE 


Figure  11  Logical  -or-  Truth  Table 
The  exclusive  -OR-  gate  logic  involves  the  input  words  (AB)  , 


00,10,01"  and  the  output  words  (C), 


0,  1,  1." 


TRUTH  TABLE 
ABC 


0  1  0 
1  0  0 
1  \  1 


C  -  A  +  B 


Figure  12  Logical  -and-  Gate  Truth  Table 

The  -AND-  gate  logic  involves  input  words  (AB),  "00,  01,  10, 
11"  and  output  words  (C),  "0,  0,  0,  1."  Figure  13  shows  the  truth 
table  for  the  logical  gates  used  in  the  critical  task  decision 
tree. 


TRUTH  TABLE 
ABC 

0  0  0 
0  1  1 
1  0  1 


AUTOMATIC 

(A) 


MANUAL 

(B) 


Figure  13  Logical  -or-  Truth  Table 
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The  exclusive  -OR-  gate  logic  involves  the  input  words,  (AB), 

•WL.* 

"00,  10,  01"  and  the  output  words  (C),  "0,  1,  1." 

9.0  PILOT-AVIONIC  TASK  MISSION  RISK  AND  BENEFIT  ASSESSMENT 

There  should  be  an  optimal  balance  between  risks  and  benefits  in 
every  mission.  However,  before  an  optimal  balance  can  be  arrived 
at,  there  must  be  a  risk  and  benefit  evaluation  to  the  degree  of 
accuracy  required  by  the  parameters  of  the  decision  process. 

The  principle  of  risk  as  a  potential  loss  is  conceptually  sim¬ 
ple;  however,  the  assessment  of  risk  is  complex  due  to  the  number  of 
comprizing  elements.  A  risk  assessment  model  should  have  the  fol¬ 
lowing  minimum  elements  as  shown  in  Figure  14. 


Figure  14  Risk  and  Benefit  Assessment  Model 


Potential  basic  events  are  low-order,  causal  events  that  may 
prevent  the  occurrence  of  the  head  event.  Potential  head  events 
are  a  set  of  top-level 'mission  task  flow  events.  The  implication 
for  risk  assessment  is  that  one  must  consider  a  number  of  top- 
level  events  that  may  be  involved.  A  potential  head  event  may  be 
thought  of  as  the  critical  event  for  the  accomplishment  of  the 
mission  task  flow.  Basic  events  include  all  of  the  man-machine 
interface  actions  required  to  accomplish  the  head  event. 

The  potential  consequences  are  the  immediate  results  of  the 
nonoccurrence  of  the  head  event.  These  consequences  are  dependent 
upon  mission  circumstances  and  are  highly  variable.  However,  po¬ 
tential  consequences  can  describe  the  achievement  losses. 


30 


260 


The  probable  benefits  place  a  value  on  mission  accomplishment. 
Unfortunately,  there  is  always  a  probable  loss  to  the  mission 
achievement,  such  as  missing  a  navigation  way  point.  The  assessment 
raises  a  number  of  questions  such  as,  how  to  determine  the  poten¬ 
tial  loss  to  the  mission  by  missing  a  navigation  way  point,  or  by 
missing  a  target  of  opportunity?  If  the  severity  of  the  potential 
loss  is  great,  such  as  missing  a  mission  objective  target  or  by 
loss  of  an  aircraft,  then  the  assessment  is  less  difficult. 

In  the  final  analysis,  judgement  as  to  "acceptability"  of  risk 
is  always  highly  subjective.  However,  the  criteria  required  to  set 
subjective  decision  points  must  be  objective  in  nature  and  present 
information  to  help  balance  the  benefits  against  the  risks. 

Table  6  illustrates  the  critical  task  probability-risk-severity 
relationship  that  should  be  considered  in  the  risk  and  benefit  as¬ 
sessment  . 


Table  6 


PROBABILITY-RISK-SEVERITY  RELATIONSHIP 


Probability 

Risk 

Severity 

1.  Impossible 

6 .  None 

6.  Nill 

2.  Improbable 

5.  Very  small 

5.  Nuisance 

3.  Small 

4.  Small 

4.  Marginal 

4.  Significant 

3.  Significant 

3.  Significant 

5.  High 

2.  High 

2.  Extreme 

6.  Extreme 

1.  Extreme 

1.  Catastrophic 
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Figure  15  shows  the  relationship  of  probability,  risk,  and 
severity  in  diagramatic  form.  By  cross  referencing  Table  6  and 
Figure  15,  it  should  be  noted  that  when  the  risk  is  extreme  (1)  and 
the  severity  is  catastrophic  (1),  the  probability  is  impossible  (1). 
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Figure  15  Probability-Risk-Severity  Relationship 
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GLOSSARY  FILE 


NOMENCLATURE  C  N] 

ON  C/l 

ARMAMENT  "O O E/ D I SPL A T  PANEL  CAMDl 


A-A>'N/ AMD 
A-G>“N/ AMD 
ARM>~N/ AMD 
9M3>*N/ AMD 
3M3  S  W>” N /  A  M  D 
9M8  SW-DUAL"N/ A'«D 
3M8  SW-$N5L"N/ AMD 
OEO'N/ AMD 
DED>“N/ AMD 
DED  S3C>“N/AMD 
DED  S3C-DEC>*N/ AMD 
DED  S9C-INC>"N/ AMD 
DUAL>"N/AMD 
GUN>"N/ AMD 
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GUN  SW-SA  F>"N/AMD 
HUD>*N/ AMD 
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HUD  SBC-DEC>"N/ AMD 
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INO’N/ AMD 
LMFD>"N/AMD 
LMFD  SBC>~N/ AMD 
LMFD  SeC-DEC>"N/AwO 
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MSL>“N/ AMD 
MSL  SW>"N/AMO 
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43*1 -*OVE  3"3  $W  TO  DOUBLE  30*6  RELS43S  POSITION 
4  3*1  2.5/  5.0/  0.0/  Q 

A3M2-*CVE  3*3  S V  TO  SINGLE  30*0  RELEASE  POSITION 
A9M2  2.5/  5.0/  0.0/  0 

43M9-»0SITICN  3*3  DU4L/SN3L  SW  ON  4*D  AS  DESIRED 
A3M3  2.5/  5.3/  0.0/  2  0  A3, *143*2 

A0E1-P0TATE  DEO  S9C  KNOB  TO  SYM3CLCGY  DECREASE  INTENSITY  POSITION 
A  0  c 1  2.5/  5.0/  0.0/  0 

AOE 2“ ROTATE  OED  S9C  <NC3  TO  SY NEOLOGY  INCREASE  INTENSITY  POSITION 
A  DE  2  2.5/  5.0/  0.0/  0 

ADED” POSITION  DED  S3  C  <N03  ON  A*D  AS  DESIRED 
ADED  2.5/  5.0/  0.0/  2  0  AD61ADS2 

ASU1-N0VE  GUN  SW  TO  GUN  IN  ARM  MODE  POSITION 
A  GU 1  2.5/  5.0/  0.0/  0 

AGU2-*0VE  GUN  SW  TO  GUN  IN  SAFE  MODE  POSITION 
A  GU  2  2.5/  5.0/  0.0/  0 

AGUN— POSITION  GUN  4RM/S»F  SWITCH  ON  AMD  AS  DESIRED 
AGUN  2,5/  5.0/  0.0/  2  0  AGU14GU2 

A HU 1“R0T ATE  HUD  S 3 C  <N03  TO  SYM30L0GY  DECREASE  INTENSITY  POSITION 
A  HU  1  2.5/  5.0/  0.0/  0 

AHU2-R0TATS  HUD  SBC  KN09  TO  SYN30LOGY  INCREASE  INTENSITY  POSITION 
A  HU  2  2.5/  5.0/  0.0/  0 

AHUD— POSITION  HUD  SEC  KNQ9  ON  AMD  AS  DESIRED 
A  HU  0  2.5/  5.0/  0.0/  2  0  AHU1AHU2 

ALM1 -ROT ATc  L*F  D  S3C  KNOB  TO  SYMBOLOGY  DECREASE  INTENSITY  POSITION 
ALM1  2.5/  5.0/  0.0/  0 

AL. M2— ROTATE  L*FD  S3  A  KN03  TO  SYM30LOGY  INCREASE  INTENSITY"  POSITION 
AL*2  2.5/  5.0/  0.0/  0 

ALMF- POSITION  LMFD  S  3C  KNOB  ON  AMD  AS  DESIRED 
ALMF  2.5/  5.0/  0.0/  2  0  ALM1  AL.M2 

AMS1-M0VE  MSL  SW  TO  MISSILE  FIRING  IN  PAIR  POSITION 
A  MS  1  2.5/  5.0/  0.0/  0 

AMS  2- MOVE  .MSL  SW  TO  SINGLE  MISSILE  FIRING  POSITION 
A  ms  2  2.5/  5.0/  0.0/  0 

A  *S  L-PO  S IT  I  ON  MISSILE  PAIR/S NGL  SWITCH  ON  A*D  AS  DESIRED 
A*SL  2.5/  5.0/  0.0/  2  0  AMS1AMS2 

AR*1  — ROTATE  R * F D  S 3 C  KNO  =  TO  SYMBOLOGY  INCREASE  INTENSITY  POSITION 
ARM1  2.5/  5.0/  0.0/  0 

ARM2-R0TATE  R  *  F  D  S3C  KNOB  TO  SYM30L0GY  DECREASE  INTENSITY’  POSITION 
A  R M 2  2.5/  5.0/  0.0/  0 

AR.MF—  POSITION  R  *  F  0  S3C  KN03  ON  AMD  AS  DESIRED 

ARMF  2.5/  5.0/  0.0/  2  0  A  R  M 1 ARM 2 

HA-A'-OBSEPVE  HUD  TO  VERIFY  A-A  MNEMONIC 

HA-A  0.5/  3.5/  0.0/  0 

HA-G-OBSEP'/S  HUD  TO  VERIFY  4-3  MNEMONIC 

HA-G  0.5/  3.5/  0.0/  0 

HAR*— 03SERVE  HUD  TD  VERIFY  ARM  MNEMONIC 

HARM  0.5/  3.5/  0.0/  0 

H3M9-03SERVE  HUD  TO  VERIFY  3M9  MNEMONIC 

H9M9  0.5/  3.5/  0.0/  0 

HOU A-03  SERVE  HUD  TO  VERIFY  DUAL  MNEMONIC 

H  DU  A  0.5/  3.5/  0.0/  0 

H5UN- OBSERVE  HUD  TO  VERIFY  GUN  *NEM0NIC 

H  GU  N  0.5/  3.5/  0.0/  0 

H MSL-OB  SERVE  HUD  TO  VERIFY  MSL  *NEM0NIC 

HM3L  0.5/  3.5/  0.0/  0 

HNAV- OB  SERVE  HUD  TO  VERIFY  NAV  MNEMONIC 

HNA V  0.5/  3.5/  0.0/  0 

HPAI-OBSHRVE  HUD  TO  VERIFY  PAIR  MNEMONIC 


H  =» A  I  0.5/  3.5/ 

4SAF-0ESERVE  HU? 
M S  A  c  0.5/  3.5/ 

HSNG-08SERVE  «U0 
HSNG  0.5/  5.5/ 

N4-A-08S£RVE  ^FD 
MA-A  0.5/  3.5/ 


3.0/  0 

T  3  VERIFY 
0.0/  0 
TO  VERIFY 
0.0/  0 
TO  VERIFY 
0.0/  0 


SAF  MNEMONIC 
SNGL  MNEMONIC 
A-A  MNEMONIC 


MA-G-03S£RVE  “FD 
MA-G  0.5/  3.5/ 

M  A  R  **-  0  3  S  E  R  V  E  "FO 
MAR'-  0.5/  3.5/ 

M3M3-03SSRVE  «F0 
M9M3  0.5/  3.5/ 

MDU  A— 06  S  £  5  V  E  MFD 
MDUA  0.5/  3.5/ 

MGUN-03$=RVE  VF  D 
MGUN  0.5/  3.5/ 
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0.0/  0 
TO  VERIFY 
0.0/  0 
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0.0/  0 
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0.0/  0 
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A-G  MNEMONIC 
ARM  MNEMONIC 


SMB  MNEMONIC 


DUAL  MNEMONIC 
GUN  MNEMONIC 
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0.5/  3.5/  0.0/  0 
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0.5/  3.5/  0.0/  0 
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0.5/  3.5/  0.0/  0 
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PNAV  1.5/  4.0/  0.0/ 
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rocedure 


for 


access i ng 


computers 


through 


the  EDWARDS  TAC . 


1.  Make  sure  your  terminal  and  modem  are  on  and  happy. 

2.  Call  one  of  the  TAC  numbers  listed  below: 

277-7949 

277-7966 

277-7968 

277-7970 

277-7971 

277-7972 

277-7973 

277-7974 

3.  Once  the  TAC  answers  and  your  modem  is  connected  with  it,  type  a 

"CONTROL-Q".  You  do  this  by  holding  down  the  key  marked  "CTRL" 
and  simultaneously  hitting  the  "0"  key. 

4.  The  TAC  will  then  respond  with  a  sign-on  message,  and  await  your 

command . 


Depending  on  the  computer  you  want  to  connect  to,  type: 

@o  26.1.0.39  (for  the  Edwards-2060) 

@o  26.0.0.39  (for  the  Edwards-VAX) 

@o  67  (for  the  AFSC-HQ  at  Andrews) 


6.  The  TAC  will  then  .prompt  for  USER  ID  and  ACCESS  CODE.  Enter  these  as 

they  are  written  on  your  TAC  ACCESS  CARD. 

7.  You  will  then  see  some  "progress  of- connection"  messages. 

8.  Hit  <RETURN>  a  couple  times  to  get  the  destination  computer's  attention. 

9.  Once  the  destination  computer  responds,  you  may  log  in  just  as  if  you 

were  at  the  machine's  site. 


10.  When  you  are  done  on  the  destination  computer,  type  "LOGO"  to  logout. 
You  will  be  returned  to  the  Edwards  TAC,  where  you  may  point  at  another 
destination  computer,  or  simply  type  "@1"  to  log  out  of  the  TAC. 

11.  Hang  up,  and  you're  done... 


SESSION  3.  PHYSIOLOGICAL  TECHNIQUES 
Presenters 

G.  F.  Wilson:  Neuropsychological  Workload  Test  Battery 


J.A.  Stern:  Workload  -  A  Psychophysiological  Approach 


ID-A183  638 


UNCLASSIFIED 


PROCEEDINGS  OF  THE  DOD  WORKLOAD  ASSESSMENT  WORKSHOP  ON 
WORKLOAD  ASSESSNEN.  .  <U>  NAVAL  UNDERWATER  SVSTEHS  CENTER 
NEWPORT  RI  HD  FIEDLER  13  SEP  87  NUSC-TD-6688 


cc 

O 

£ 


< 

o 

o 

o 


(0 

CL 

o 

cc 


<  UJ 

Jr  cd 


CD  < 
CD 


CD 

o  UJ 


1  g 
o  o 


O  cc 

O  2 
CC  ^ 
Lii  Z 


Q  O 
5  CD 


<  S 

Q  H- 

<  § 

O  £ 


*  t 

CC  x 

O  C5 

8 1 


?F  Xf° 

°1  °fe 

_T  -,’XP 


§  §5 

s- 

co 

Z>  g 
ZkI  S 

UJ5S2  1 
Ij  <tr  ro 

CD<^ 


.  o . - ^  ^  .Vj awmms 


HSfts? ; 


o  P 

_J  z 
o  > 

O  £ 

O  f- 

O  ^ 
ICQ 

O 


2  <  h 
<  Q  h 

O  CL  < 
C/D  D  1U 


LLl 

S  o 


_j  C/D  LLl  “  LU  /P! 

j  v  v  >  Q[  h  .5 

<  OC  QC  H  O  § 

9  o  o  o  t  Z  ^ 

1  -  _  I  I  I  _  I  1 


_c9oootz^i 

—  "S  LU  ^  —  O  c/D 

ffl  CD  ^  ^  I  ^  ^  <3* 

Q  LU  LU  ill.O  I  5  5 

055c/)2cdkil 


Q 

LLl 

^  Q 
O  CC 
>  < 
LLl  O 

LU  CD 

!<  DC 
p  LU 
CO  * 
>  0 
Q  £ 

LLl  LJ 


<  Q 

O  LU 


UJ  J 
2  z 

CO  D 


LU 

*: 

z 

1- 

g 

LU 

_J 

_l 

CO 

DC 

O 

LU 

< 

CO 

>~ 

LU 

3 

LU 

X 

• 

• . 

• 

(/) 

-JLIJ 

<5 

CC  ^ 
hW 

z  < 

02 


<co 

CC  LU 
LU  CC 
ID 
CL  CO 

GC^ 

Slu 


<co 

siu 

cctt 

£| 

a!  2E 


vs 


WHKScwSc 


wif- 


COMPONENTS  OF  THE  NWTB 


288 


TEST  BATTERY  MENU 


289 


UJ 

X 


6 

I 

E 


X 

X 

X 

• 

X 
X 
X 
X 
X 
X 
•  • 

o 

a 


a 


o 

z 


rr  01 

H 

O 


h- 

z 

LU 

> 

LU 

LU 

EC 

< 

oc 

o 

a 


LU 

CO  LU 

2  & 

2s 
CO 
LU 
CC 


Z 


Q 

LU 

* 

o 


co 

>- 
Q 
< 
LU 
I — 
CO 

>- 


> 

LU 

o 

CO 

CO 

* 

LU 

z 

CO 

LU 

LU 

h- 

o 

UJ 

X 

LU 

LU 

£ 

LU 

X 

CO 

>- 

h- 

o 

(— 

CO 

LU 

h- 

< 

X 

< 

1— 

w 

_J 

X 

LU 

CL 

< 

CO 

LU 

< 

UJ 

\r~ 

_l 

CL 

>- 

X 

z 

CO 

CO 

Z 

< 

_J 

Q 

UL 

< 

z 

z 

_J 

o 

Z> 

CO 

CL 

< 

LU 

< 

LU 

1— 

I 

O 

a 

X 

LU 

< 

X 

ao 

LU 

>- 

1- 

X 

> 

X 

CO 

X 

LU 

CO 

X 

LU 

o 

c\i 

CO 

tri 

CD 

ad 

03 

i- 

UJ 

O 


CO 

LU 


CD 


tr 

O 

LL 

a: 

LU 

0_ 

co 

3 

O 

Z) 

z 


CNJ 


13.  SELECTIVE  ATTENTION 

16.  START  TESTS 

17.  CHANGE  FILE  ID 

ENTER  NUMBER  OF  SELECTED  REQUEST 
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DEFINE  WORK  LOAD 

IT  CANNOT  BE  DEFINED  SOLELY  ON  BASIS  OF: 

1.  TA§K  DEMANDS  -  Input  requirements 

-  (Domain  of  engineer) 

2.  PROBLEM  SOLVING  STRATEGY  -  Decision  making 

requirements 

-  (Domain  of  psychologists) 

3.  RESPONSE  COMPLEXITY  -  Output  requirements 

-  (Domain  of  human  factors) 


WORK  LOAD  IS: 

The  resultant  of  the  interaction  between: 

1.  TASK  DEMANDS 

2.  PERFORMER  ATTRIBUTES 

3.  RESPONSE  REQUIREMENTS 
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WORK  LOAD 

AVERAGE  -  Integrated  over  time  -  MINUTES,  HOURS 
MOMENTARY  -  Integrated  over  time  -  SECOND (S) 


OUR  FOCUS  -  on  MOMENTARY 

WHY: 

1.  TASK  DEMANDS  -  generally  vary  over  time 

2.  PERFORMER  ATTRIBUTES  -  generally  vary  over  time 

MOTIVATION,  ATTENTION,  etc.,  FLUCTUATES 

3.  RESPONSE  REQUIREMENTS  -  vary  over  time 


*  / 
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How  long  is  a  MOMENT 

Depends  on  question  being  asked 

Work  load  associated  with  touch  down 

10  -  60  seconds 


b. 


and  roll-out  of  aircraft 


Work  load  concern  with  HUMAN  ERROR 

.1-30  seconds 


HUMAN  ERROR  -  leads  to  ACCIDENT  -  some  of  the  time 

CO-OCCURRENCE  of 

1.  HIGH  TASK  DEMANDS 

Such  as  unusual  environmental  event 

2.  EQUIPMENT  PROBLEM 

Such  as  blow-out  of  tire 

3.  HUMAN  PROBLEM 

Attention  -  Distracted 

-  Fatigued 

-  Drugged 

-  etc. 

A  MOMENT  OF  LOWERED  ATTENTION  coupled  with  MOMENT  of  HIGH  TASK 
DEMANDS  and/or  EQUIPMENT  PROBLEMS  results  in  HUMAN  ERROR  which 
can  result  in  ACCIDENT. 


’*  "*  Oj  'j'v  v 
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Our  strategy  for  using  physiological  measures  in  work  load 
assessment  - 

1.  Laboratory  studies  to  define  physiological 
parameters  that  may  reflect  ’components  of 
work  load  (attention,  drowsiness,  distracted, 
etc. ) 

-  RELATE  physiological  measures  to 
PERFORMANCE  measures 

-  DEVELOP  predictive  equation  allowing 
for  use  of  physiological  measure  to 
predict  alterations  in  PERFORMANCE 

-  VALIDATE  equation  in  a  number  of 
laboratory  settings 

2.  Apply  in  SIMULATION  or  REAL  WORLD 


3.  Develop  hardware/software  for  real  time 
implementation  of  measures. 


What  can  physiology  contribute  to  WORK  LOAD  assessment 


1.  It  is  resultant  of  interaction  between 

A.  TASK  DEMANDS 

B.  PERFORMER  ATTRIBUTES 

C.  RESPONSE  REQUIREMENTS  -  as  recorded 

from  the  performer 

2.  It  can  be  sampled  continuously 

3.  It  can  be  sampled  surreptitiously  without 
interfering  with  task  performance 

4.  It  is  objective  -  not  dependent  on  subject 
self-report. 
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Lt.  Col.  W.  L.  Derrick  and  Lt.  Col.  R.  M.  McCloy,  USAF 
Behavioral  Workload  Assessment  Techniques 


BEHAVIORAL  WORKLOAD 
ASSESSMENT  TECHNIQUES 


(Secondary  Task  Methodology) 

William  L.  Derrick  &  Thomas  M.  McCloy 
United  States  Air  Force 

Sources  for  this  Presentation: 

1)  Handbook  of  Perception  and  Human  Performance  (K.  Boff  &  Kaufman, 
Eds.) ,  1986. 

a.  Gopher,  D.  &  Donchin,  E.  "Workload  —  An  Examination  of  the 
Concept",  especially  sections  1,  2.1,  2.5  -  2.8,  and  3-3.8. 

b.  O'Donnell,  R.  &  Eggemeier,  T.  "Workload  Assessment  Methodology", 
especially  sections  1,  3,  and  4. 

2)  Engineering  Psychology  and  Human  Performance,  C.  D.  Wickens,  1984, 
especially  chapters  1  and  8. 

You  should  read  these  references.  If  you  have  little  or  no  background  in 
behavioral  science,  start  with  the  O'Donnell  and  Eggemeier  chapter.  The  Gopher 
and  Donchin  chapter  is  well  written  but  it  deals  with  some  complex  concepts. 
The  Wickens  text,  which  discusses  the  research  and  theories  the  other  two 
readings  draw  upon,  is  graduate- level  material. 

You  should  also  stay  current  in  the  literature.  The  Proceedings  from  the  Annual 
Meeting  and  Human  Factors  have  several  articles  on  workload.  But  remember, 
almost  anyone  can  publish  anything  in  the  Proceedings,  so  some  questionable  work 
nay  be  found  there.  Also,  Human  Factors  may  have  articles  that  are  difficult  to 
understand,  deal  with  very  theoretical  (and  esoteric)  concepts  that  will  not 
help  a  practitioner,  and  may  contain  work  of  a  so-so  quality  even  though  the 
articles  receive  a  peer  review.  Keep  in  mind  that  technical  reports  frcm 
laboratories  and  contractors  may  not  receive  much  of  a  technical  review  either. 

I.  Why  don't  we  have  a  "How-to”  workload  quide  for  secondary  tasks? 

*  Wierwillie  and  Williges  published  guides  in  1978 


*  The  Boff  and  Kaufman  volume  is  as  close  as  we  ccme  in  1986. 


*  Reason:  Several  theoretical  issues  are  still  being  resolved 
(and  some  may  never  be) . 


*  Any  guide  (or  research)  that  ignores  these  issues  is  snake  oil. 


II.  Must  start  with  a  definition  of  workload.  This  will  guide  the  selection 
of  secondary  tasks,  their  method  of  employment,  and  interpretation  of 
results. 

*  Still  won't  find  complete  agreement  on  the  definition,  but  we've  made 
much  progress. 
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Can't  define  workload  as  just  task  difficulty.  Any  method  of  task 
description  does  not  account  for  different  people  performing  the  task 
at  different  times  in  combination  with  other  sets  of  activities.  Any 
one  task  characteristic  can  produce  a  whole  range  of  "workload" 
results. 

Can't  just  check  system  performance.  Very  different  system  designs  can 
lead  to  equivalent  criterion  performance  but  intuitively  we  know  their 
"workloads"  are  different. 


Can't  just  ask  operators.  Raters  may  be  unreliable,  have  biases, 
recall  difficulties,  and  poor  (or  no)  insight  into  what  produced  the 
workload. 


*  Therefore,  let's  define  workload  as  the  interaction  between  the 
operator  and  the  task.  When  the  demands  of  the  task  exceed  the 
operator's  capability  to  deliver,  we  invoke  the  hypothetical  construct 
of  workload  (or  overload) . 


III.  What  do  we  ignore  in  this  workload  definition? 
*  motivation;  assumed  to  be  adequate. 


*  learning;  the  skills  to  perform  the  task  are  in  the  operator's 
repertoire. 


*  ability;  if  the  operator  does  not  possess  the  needed  ability(ies) , 
workload  cannot  explain  the  results. 


IV.  What  operator  capabilities  must  we  consider? 

*  "Mental"  workload  suggests  mental  capabilities. 


*  Long  and  rich  history  of  studies  in  experimental  psychology  suggest 
that  we  can  conceptualize  an  information  processing  system  between 
task  stimulus  and  operator  response. 


*  This  system  has  an  architecture  (e.g.,  perception,  memory,  response 
selection)  and  an  energetical  component.  Terms  such  as  attention, 
mental  energy,  processing  resources,  processing  capacity  refer  to  the 
latter . 


"mental  workload  may  be  viewed  as  the  difference  between  capacities 
of  the  information-processing  system  that  are  required  for  task 
performance  to  satisfy  performance  expectations  and  the  capacity  that 
is  available  at  any  given  time."  (Gopher  &  Donchin)  This  is  a 
closed-loop  definition. 


E 
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*  Task  difficulty  may  now  be  defined  by  amount  of  resources  demanded. 

A  "different”  task  demands  most  of  the  resources  available;  a  "very 
difficult"  or  "overly  difficult"  task  demands  more  resources  than  are 
available  at  that  instant  in  time. 


V.  What  are  some  of  the  characteristics  of  processing  resources? 

*  the  fuel  that  makes  the  information-processing  system  operate. 

*  limited  in  quantity  at  any  point  in  time. 

*  probably  not  expanded  by  arousal. 

*  can  be  allocated  to  different  parts  of  the  architecture. 


* 


not  the  same  thing  as  the  laymen's  concept  of  attention;  not  open  to 
introspection  (thus  a  problem  if  subjective  measures  are  used) . 


*  experimental  data  suggests  more  than  one  type  or  pool  of  resources  (this 
complicates  workload  assessment  considerably) . 


*  not  possible  to  observe  so  must  make  inferences  based  upon  task 

performance;  however,  lots  of  possible  reasons  for  any  given  level  of 
task  performance. 


s 

i 

v 
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*  research  suggests  that  certain  highly  skilled  tasks  with  specific  input 
and  response  characteristics  may  demand  almost  no  resources  at  all  even 
though  the  task  was  resource  demanding  at  one  time  (an  "automatic" 
process) . 


VI.  So  what  does  all  this  have  to  do  with  the  use  of  behavioral  techniques  in 
the  assessment  of  workload? 

*  We  desire  to  know  the  composite  resource  demands  of  the  task  of 

interest.  We  will  call  this  workload  and  note  that  it  is  a  function 
of  an  interaction  between  the  operator  and  the  task. 


Resource  demands  can  be  inferred  only  from  the  results  of  task 
performance. 


The  task  of  interest  most  likely  has  multiple  resource  demands.  We 
must  uncover  the  resource  load  profile.  Wickens  states  that  a  load 
profile  will  consist  of  three  dimensions: 

1)  stages  of  processing:  perception  &  central  processing  vs. 
responding. 

2)  codes  of  central  processing:  verbal  vs.  spatial. 

3)  modalities  of  input  and  response:  auditory  vs.  visual  input; 
manual  vs.  vocal  response. 


We  look  at  the  primary  task  and  make  some  guesses  as  to  what  resources 
are  demanded  for  acceptable  task  performance.  We  then  choose  a 
battery  of  secondary  tasks  that  are  known  to  demand  specific  types  of 
resources.  If  we  are  lucky,  we  know  the  ordinal  amount  of  resources 
demanded  by  these  secondary  tasks  (little,  some,  a  lot) .  The  primary 
tasks  must  interfere  on  some  of  the  resource  dimensions. 


When  tasks  "interfere"  they  demand  the  same  resources  for  performance. 
If  the  joint  resource  demand  is  great  enough,  sortie  degredation  in 
primary  or  secondary  task  performance  (or  both)  should  occur.  This 
will  permit  you  to  estimate  what  type  and  roughly  what  level  of 
resources  are  demanded  by  the  primary  task. 


In  general,  this  approach  is  diagnostic.  It  tells  you  what  types 
and  what  levels  of  resources  are  demanded  by  the  primary  task  when 
performed  to  criterion.  Thus,  you  obtain  a  resource  load  profile 
(workload  profile)  on  three  dimensions. 


More  specifically,  what  steps  are  employed? 

1)  The  task  of  interest  is  isolated  and  analyzed  for  potential  resource 
demands. 

2)  Operator  motivation  is  established  at  the  appropriate  level  for  the 
user  population. 

3)  The  task  is  learned  by  the  operators  until  performance  reaches  a 
plateau  or  sortie  level  of  stability.  Good  luck. 

4)  The  battery  of  secondary  tasks  is  selected.  Hopefully  none  are 
locally  developed.  These  tasks  should  demand  some  of  the  same  resources 
as  the  primary  task  and  those  resource  demands  should  have  been 
previously  validated. 

5)  Each  secondary  task  is  practiced  until  a  stable  level  of  performance 
is  reached. 

6)  The  primary  and  secondary  tasks  are  performed  jointly.  Each 
secondary  task  should  have  some  aspect  that  interferes  with  the  primary 
task  but  does  not  intrude. 
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7)  Operators  are  given  the  instruction  that  the  primary  task  should 
take  priority;  i.e.,  its  performance  in  the  dual  task  mode  should  equal 
performance  in  the  single  task  situations  (but  see  below) . 

8)  Single-to-dual  task  performance  changes  for  each  secondary  task  are 
examined. 

9)  A  pattern  of  performance  decrements  and  no  decrements  should  emerge. 
This  pattern  can  be  translated  into  a  load  profile.  For  example,  you 
might  conclude  that  your  primary  task  has  the  following  load  profile 
(e.g.,  workload) : 

a.  Some  demand  for  perception  and  central  processing  resources 
and  a  large  demand  for  response-execution  resources. 

b.  Some  demand  for  both  verbal  and  spatial  resources. 

c.  No  demand  for  auditory  resources  but  a  very  heavy  demand 
for  visual  resources.  Saties  demand  for  manual  but  a  very  small 
demand  for  vocal  response  resources. 

VIII.  What  might  produce  nearly  perfect  dual  task  performance? 

*  The  composite  set  of  resource  demands  in  the  primary  task  has  nothing 
in  cannon  with  the  resource  demands  in  the  secondary  task.  Unlikely 
if  you  properly  analyzed  your  primary  task  and  chose  a  validated 
secondary  task. 


*  The  resource  demands  of  the  primary  or  secondary  tasks  (or  both)  were 
"data  limited"  (translation:  very  small  separate  demands  so  that  the 
combined  demand  could  easily  be  accanodated  by  the  operator) .  A  real 
possibility.  If  your  secondary  task  is  data  limited,  you  do  not  have 
a  diagnostic  instrunent. 


IX.  Where  do  I  get  my  hands  on  these  secondary  tasks? 

*  If  you  have  the  theoretical  background,  the  laboratory  and  equipment,  a 
software  person,  and  the  tine,  you  can  create  them  yourself.  Read 
about  the  tasks  used  in  the  resource  model  literature  and  build  them 
for  your  use.  Much  can  go  wrong  here.  We  don't  recommend  this 
approach  for  the  practitioner  with  a  vforkload  answer  to  report  in  a 
relatively  short  period  of  time. 


*  A  criterion  task  battery  is  being  developed.  Same  papers  will  be 
reported  at  this  conference.  You  may  be  able  to  get  enough  detail  from 
the  authors  to  create  these  tasks  for  your  own  use.  They  may  be 
available  for  your  use  if  you  provide  the  data  to  the  researchers. 

*  Consultants  are  always  out  there  but  be  careful.  You  don't  have  to 
understand  anything  on  the  proceeding  pages  to  call  yourself  a 
consultant. 


*  °°  not  use  a  set  of  tasks  you  happen  to  have  leftover  from  previous 

work  just  because  they  are  available.  They  must  be  validated  as  to 
the  types  and  ordinal  levels  of  resources  demanded. 


How  should  dual-task  performance  decrements  be  measured? 

*  The  answer  isn't  an  easy  one  to  give.  Debate  exists  as  to  how  to  scale 
the  changes  in  performance  from  single  to  dual  task  and  how  many  data 
points  are  needed  for  each  primary-secondary  task  pair. 


*  One  procedure,  developed  by  Wickens  is  based  upon  the  normalized  score 
concept  (score-mean/standard  deviation).  For  any  task,  the  mean  *  the 
operator's  single  task  performance,  the  score  =  the  operator's 
performance  of  that  task  when  in  the  dual  task  mode,  and  the  standard 
deviation  =  the  single  task  performance  standard  deviation.  This 
procedure  produces  a  decrement  score  that  can  be  compared  to  a  single 
task  score.  Decrement  scores  for  all  tasks  can  be  plotted  for  visual 
comparison  in  what  has  been  called  a  POC  (Performance  Operating 
Characteristic)  space.  (However,  this  is  not  a  true  POC  -  see  below). 
Statistical  tests  (very  complicated  ones)  can  compare  all  dual  task 
points  to  determine  if  the  decrements  are  significant  (See  Wickens 
et.al.,  1980,  in  Human  Factors  and  Derrick,  1981,  in  Proceedings 
for  examples) . 


*  Another  approach  is  simply  to  scale  performance  on  the  metric 

appropriate  for  each  task  (e.g.,  task  1  -  reaction  time,  task  2  -  RMS 
error) .  Changes  within  each  task  can  be  evaluated  statistically, 
but  evaluation  of  joint  task  performance  is  not  clear  since  you  have 
the  old  apples  vs.  oranges  problem  (See  Gopher  et.al.,  1982  in 
Journal  of  Experimental  Psychology:  Human  Perception  and 
Performance,  and  our  article  in  the  1984  Proceedings) . 


What  do  you  mean  by  number  of  data  points  in  X  above? 

*  A  very  good  case  can  be  made  against  the  classic  dual  task  methodology 
considered  thus  far.  Recall  that  operators  are  asked  to  guard  or 
protect  primary  task  performance  in  the  dual  task  situation.  The 
purpose  here  is  to  index  the  types  and  levels  of  resources  demanded  for 
normal  primary  task  performance  with  performance  changes  (or  lack 
thereof)  in  the  battery  of  secondary  tasks.  This  is  probably  a  naive 
approach.  First,  as  the  dual  task  is  practiced,  strategies  may  be 
developed  to  integrate  the  separate  task  components,  so  the  joint 
resource  demand  is  less  than  the  sum  of  the  separate  resource  demands. 
Second,  a  type  of  dual  task  interference  may  arise  that  produces  a  drop 
in  performance  that  has  nothing  to  do  with  the  true  resource  -  workload 
question  (e.g.,  increased  visual  scanning).  Wickens  originally  termed 
this  structural  interference.  Third,  operators  may  not  follow  the 
instructions  to  guard  primary  task  performance;  it  may  drop  or  even 
increase,  and  you  are  stuck  with  a  secondary  task  score  that  in  and  of 
itself  gives  you  erroneous  informat  ion. 


The  answer  to  these  problems  is  the  true  POC,  not  the  single  point  POCs 
you  will  find  in  Wickens  et.al.,  1980  and  Derrick,  1981,  Gopher  et.al., 
1982,  first  reported  this  although  ours  (1984)  utilizes  a  better 
methodology.  Here  operators  are  told  to  allocate  X  percent  of  their 
"attention",  a  term  they  can  understand,  to  one  task  and  1-X  percent  to 
the  other.  At  least  three  different  allocation  percentages  per  dual 
task  are  used.  The  resultant  data  produces  same  type  of  curve  that 
depicts  the  whole  range  of  resource  and  performance  tradeoffs  for  a 
primary-secondary  task  pair.  Tbus  you  get  the  resource  "cost"  for  any 
given  level  of  performance,  and  that  relationship  may  be  anything  but 
1 inear . 


POCs  are  all  well  and  good  theoretically,  but  they  are  very  difficult 
to  obtain.  The  allocation  policy  instructions  require  the  clever  use 
of  on-line  performance  feedback  if  they  are  to  be  followed.  To  get  the 
needed  data  from  an  operator  requires  3  to  5  times  the  laboratory  time 
for  each  primary  -  secondary  task  pair  of  interest. 


Will  operators  accept  the  use  of  secondary  tasks  to  assess  workload? 

*  The  evidence  thus  far  isn't  good.  First,  how  can  you  describe 

workload  in  terms  of  processing  resources  to  an  operator  v*iich  must  be 
done  to  justify  the  method?  It  sounds  like  magic  or  nonsense. 

Second,  the  types  of  tasks  themselves  have  no  "face  validity"  for 
what  the  operator  considers  to  be  workload;  therefore,  expect 
resistance.  Third,  to  implement  the  required  task  battery  yourself 
takes  a  lot  of  time,  equipment,  and  special  skills.  Fourth,  to 
conduct  the  secondary  task  workload  study  requires  a  great  deal  of 
operator  time  and  skill  on  your  part.  You  should  never  attempt 
secondary  task  work  unless  you  have  been  an  apprentice.  Fifth, 
proper  analysis  of  the  data  is  by  no  means  straightforward.  Any 
given  result  may  have  many  possible  and  interconnected  causes. 


*  Embedded  secondary  tasks  have  been  proposed  as  an  alternative.  Ttie 
problem  here  is  that  what  can  be  embedded  may  well  be  unique  to  each 
workload  situation.  Thus,  the  embedded  task  may  not  be  validated 
as  to  its  resource  demands,  no  reliability  data  may  exist,  it  would  be 
difficult  to  use  an  allocation  procedure  to  get  a  POC  with  an 
embedded  task,  etc. 


So  what's  the  bottom  line  on  all  of  this? 

*  The  good  news  is  that  many  of  the  "leading  lights"  in  the  workload 
area  are  coming  to  the  position  that  workload  must  be  defined  as 
resources  consumed  during  task  performance,  they  agree  with  Wickens 
on  the  nature  of  those  resources,  they  have  identified  the  proper 
methods  of  secondary  task  workload  assessment,  and  they  are 
developing  a  validated  battery  of  secondary  tasks  to  measure 
resources. 
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The  bad  news  is  that  the  theoretical  issues  here  are  complex  and 
almost  require  graduate  training  in  cognitive  psychology,  the  process 
itself  is  time-consuming,  frought  with  peril,  and  requires  skills 
just  beneath  that  of  brain  surgery,  operators  don't  like  it,  and 
results  are  difficult  to  interpret  and  even  more  difficult  to  explain 
to  a  project  engineer. 


Should  you  give  secondary  tasks  a  chance  to  answer  your  next  workload 
question?  Only  if  you  are  willing  to  read  about  the  issues,  practice 
the  method  where  it  doesn't  count,  and  then  invest  the  time  and  money 
to  do  it  right.  Otherwise,  hire  a  consultant  with  a  track  record  of 
correct  secondary  task  workload  assessment.  If  you  choose  to 
disregard  this  approach  all  together  and  use  a  different  one, 
realize  that  other  approach  is  unlikely  to  give  you  the 
complete  workload  picture. 
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ABSTRACT 


A  crewstatlon  design  tool  Is  reviewed  which 
allows  the  human  factors  engineer  to  assess  the 
attentlonal  demands  that  will  be  Imposed  on  the 
human  operator  given  the  tasks,  times  of 
performance.  time-sharing  demands.  Individual  task 
difficulties,  and  human  Interfaces  to  be  used. 
Attention  and  performance  theories  used  In  the 
workload  model  are  discussed.  A  crewstatlon  design 
problem  example  Is  used  to  Illustrate  the  utility 
of  the  tool  In  pointing  to  automation  needs  and 
potential  reallocation  of  tasks  to  display/control 
l  surfaces. 


INTRODUCTION 


Often,  the  crewstatlon  design  team  Is  faced 
with  decisions  about  assignment  of  crew  tasks  to 
displays  and  controls  early  In  the  design  process 
without  a  method  of  evaluating  the  overall  design's 
Impact  on  the  attentlonal  resources,  and  subsequent 
workload  demands  on  the  human  opertors.  Mission 
and  task  analysis,  knowledge  of  human  performance, 
and  Initial  selection  of  generic  display/control 
areas  are  enough  to  begin  gaining  valuable  Insight 
Into  eventual  design  problems,  workload  peaks,  and 
automation  needs.  A  procedure  for  combining  these 
data  Into  an  Interactive.  Iterative  design  tool  has 
been  developed  to  provide  a  workload  profile  across 
time.  The  metric,  known  as  the  Workload  Index 
(W/INOEX).  has  been  used  on  several  recent  advanced 
design  programs,  as  well  as  for  evaluation  of 
existing  crewstatlons.  In  this  paper,  the  metric's 
features  will  be  reviewed  and  Illustrated  with  a 
design  example. 


W/INOEX  ICTHOOOLOGY 


W/INOEX  Is  a  combination  of  mission,  task,  and 
timeline  analysis,  and  theories  of  attention  and 
human  performance  used  to  predict  attentlonal 
aspects  of  workload  In  the  crewstatlon.  Previous 
m  models  such  as  Timeline  Analysis  discussed  by  Parks 
}>*y'Yl979)  Include  the  timeline  modelling  and  workload 
‘^■'•'estimation.  but  do  not  Include  a  way  to  estimate 
the  differential  effects  of  time-sharing  load 
caused  by  overlapping  resource  demands.  This 
latter  Issue  Is  the  primary  difference  between 


W/INOEX  and  other  techniques.  W/INDEX  computes  a 
level  of  workload  demand  for  each  second  of  a 
mission  by  assessing  Individual  task  difficulties 
and  the  special  problems  of  time-sharing  between 
activities.  This  computer-based  technique  operates 
on  the  following  data: 


1,  Crewstatlon  Interface  Channels  which  represent 
Input/output  of  Information  between  the  vehicle 
systems  and  the  operator.  In  Initial  crewstatlon 
design  phase,  this  list  may  represent  generic 
categories  (e.g...  visual,  manual,  verbal,  auditory) 
and  may  be  expanded  to  Include  specific 
display/ control  surfaces  as  the  design  matures. 


2.  Human  Activity  List  representing  the  present  or 
anticipated  tasks  to  be  performed  In  the  course  of 
the  mlsslon/fllght. 


3.  Attention  Involvement  Levels  which  estimate  the 
relative  attention  demand  of  each  Interface  that  Is 
required  to  perform  a  given  crew  activity.  This 
relative  scaling  Is  obtained  by  operator  opinion, 
or  any  existing  simulation  performance  data. 


4.  Interface  Conflict  Matrix  specifying  the 
time-sharing  penalties  associated  with  simultaneous 
demands  on  attention  between  two  crewstatlon 
Interface  channels  (e.g..  displays  ro  controls). 
This  Includes  problems  arising  from  suboptlmal 
placement  of  displays/controls,  and  human 
capabilities  and  limitations  In  time  sharing 
between  certain  channel  combinations.  The 
Interface  conflict  feature  reflects  models  of 
attention  and  time  sharing  known  as  "resource 
limitations"  being  researched  and  validated  by  Dr. 
Christopher  Wlckens  and  his  staff  of  the  University 
of  Illinois.  These  attention  models  reflect  the 
degradation  In  human  performance  attributed  to 
sharing  similar  resources  and  Improvement  when 
dissimilar  resources  are  shared  (Wlckens.  1984). 


5.  Operator  Activity  Timelines,  which  specify  the 
start  and  stop  times  of  each  activity  during  the 
mission  or  flight.  Times  must  be  derived  from 
existing  procedural  doctrine,  or  If  procedures  do 
not  yet  exist,  from  estimations  using  similar  types 
of  tasks.  Part  task  simulation  testing  can  be  used 
In  deriving  these  task  times. 
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The  computation  of  the  workload  demand  Index 
oparatas  on  tha  abova  data  sources  to  produca  a 
second-by-second  prof 11  a  of  demand.  Peak  demand* 
may  than  ba  axamlnad  In  mora  detail  to  determine 
possible  eorkload  alleviation  methods  (e.g., 
Information  fusion,  use  of  voice  activation  to 
offload  manual  channel,  ate.).  This  method 
provides  an  Iterative  tool  for  evaluating  the 
workload  Impact  of  cockpit  design  modifications. 

A  generic  example  based  on  an  airborne 
crewstatlon  scenario  Illustrates  the  technique. 
Table  1  lists  the  human  channels  or  Interfaces 
which  represent  generic  categories  that  can  be 
expanded  In  later  design  phases.  Table  2  shows  the 
crew  activity  list,  and  estimated  attentlonal 


Involvement  per  activity  and  display  control 
Interface.  A  fixed-wing,  single-seat  air-to-ground 
target  engagement  (anti-armor)  with  a 
"low-and-fast"  terrain  following  profile  was 
assumed  for  the  example.  Three  levels  of 
automation  are  Illustrated:  (1)  manual  flight  and 
targeting  (no  automation),  (2)  automated  target 
search/identification,  and  (B)  automated  flight  and 
target  search/identification. 

The  numbers  In  the  cells  of  the  matrix  In 
Table  2  represent  a  difficulty  Index  for  human 
performance  Indicating  an  estimate  of  the 
attentlonal  Involvement  required  by  that 
task/channel  combination,  on  a  1-5  scale.  He  have 
experimented  with  various  methods  of  obtaining  this 
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AUTOMATED  TARGETING 
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number  Including  pilot  questionnaires  with  various 
anchor  point  descriptors  (e.g.,  "simple  vs. 
complex,  "easy  vs.  difficult").  Other  techniques 
such  as  the  Subjective  Workload  Assessment 
technique  might  be  used  to  derive  this  number. 
Aldrich  and  McCracken  (1984)  propose  a  technique 
which  relates  an  Index  of  difficulty  to  specific 
task  descriptors  (e.g.«  for  cognition:  l«automat1c 
(stimulus-response);  2"Slgn.  signal,  recognition; 
3"se1ect1on  from  alternatives;  4«encod1ng/decod1ng, 
recall;  5"plann1ng  (projecting  action  sequence); 
devaluation  In  decision  making;  7«est1mat1on, 
calculation,  or  conversion. 

Table  4  Illustrates  the  conflict  matrix  of 
time-sharing  difficulty  for  pairwise  combinations 
of  the  channels  or  Interfaces.  The  scaling  of  the 
conflict  weights  Is  0.0  to  1.0  which  corresponds  to 
parallel  vs.  single-channel  performance  capability 
as  demonstrated  In  experiments  by  Wlckens, 
Mountford.  and  Schreiner  (1981)  and  others  (Navon 
and  Gopher,  1979;  Kantowltz  and  Knight,  1976;  and 
North,  1977),  and  represented  by  Wickers' 
Performance  Operating  Characteristic  (POC).  (The 
POC  shows  the  results  of  performance  when  two  tasks 
are  time-shared  In  terms  of  their  original 
sloglm-task  performance.) 

.  W/INDEX  APPLICATION 
f  V 

W/INOEX  Is  most  effective  In  several  areas  of 
the  design  process.  It  can  be  used  In  the  Initial 
stages  of  design  to  help  spot  problems  with 
operator  workload  and  find  task  elements  that  could 
benefit  from  automation.  As  the  design  matures, 
certain  tradeoffs  can  be  performed  with  the  metric 
to  check  the  effects  of  Inserting  different 
operator  Interface  technologies  such  as  speech 
command,  helmet  sighting/displays,  multi-function 
keyboards,  etc.  In  the  following  examples,  these 
uses  of  the  metric  are  Illustrated. 


Tables  1,  2  and  3  represent  tasks  and 
attention  demand  loadings  for  three  fighter  cockpit 
design  concepts.  Table  1  Illustrates  a 
non- automated,  conventional  type  of  design  defined 
as  a  "baseline."  Table  2  Illustrates  Insertion  of 
speech  command  technology  for  several  tasks  that 
were  performed  manually  In  the  baseline  (see  SPE 
column),  and  automatic  target  recognition  (ATR). 
Table  3  Illustrates  automation  of  aircraft  control 
In  the  200  ft.  above  ground  level  flight  Ingress 
phase  and  weapon  delivery  against  armored  targets. 

The  plot  In  Figure  1  Illustrates  the  three 
profiles  that  resulted  from  applying  the  W/INDEX 
metric  to  the  three  cockpit  concepts  and  the 
timeline  data.  The  attention  savings  brought  about 
by  Inserting  speech  command  and  ATR  are  evident  In 
the  dashed  line  profile  In  two  areas.  The  early 
area  between  seconds  10  and  20  are  caused  by 
shifting  the  target  data  entry  task  to  speech 
Input.  The  later  savings  In  attention  (seconds  33 
to  45)  are  due  to  automated  target  search  and 
prioritization.  A  slight  Increase  in  attention  Is 
registered  for  Cockpit  #2  In  seconds  45  through  51 
due  to  a  shift  of  demand  to  the  In-cockpit  CRT  for 
verifying  the  automated  tracking  process.  If  this 
were  shifted  back  to  the  helmet  display  concept, 
this  difference  would  disappear.  The  third  cockpit 
adding  automation  of  flight,  registers  a  profile 
about  10  points  lower  than  the  baseline  throughout 
the  segment.  Note  that  the  automation  of  flight 
does  not  completely  attenuate  workload:  Visual 
attention  Is  still  paid  to  the  aircraft's  altitude 
and  outside  world  relationships. 

The  overall  Integrated  attention  scores  for 
the  three  cockpits  were  20.1  for  the  baseline,  17.9 
for  the  cockpit  #2  concept,  and  8.7  for  the  fully 
automated  version  (cockpit  #3).  Thus  we  can 
estimate  a  "figure  of  merit"  of  pilot  attention 
demand  for  the  three  cockpits  using  the  W/INDEX 
metric. 


TABLE  3.  AUTOMATED  TF/TA  AND  TARGETING 
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Figure  1.  Comparison  of  Attention 


SUMMARY  AND  CONCLUSIONS 

W/INDEX  Is  a  flexible  tool  for  gaining  quick 
Insight  Into  the  potential  attentions!  demands  that 
different  cockpit  concepts  will  Impose  on  an 
operator/pilot.  The  metric  has  three  major  usesi 

1.  Initial  design  stages  to  Identify  automation 
needs. 

2.  Intermediate  design  stages  to  estimate  the 
effect  of  using  different  display  and  control 
concepts. 

3.  Applied  to  existing  cockpits  to  look  for  most 
severe  problem  areas  for  redesign. 

The  metric  has  the  following  advantages  over  other 
methods  of  estimating  workloads! 

1.  Rapid  reconfigurations  of  cockpit  designs  may 
be  evaluated  by  simple  additions  or  editing  of  the 
database. 

2.  Metric  considers  the  multiple-task  management 
situation  as  a  major  component  of  work1oad>  and 
uses  human  performance  and  attention  models  to 
estimate  time-sharing  difficulty. 

3.  Includes  a  cognitive  element  In  estimating 
workload  demands  taking  Into  account  decision 
making#  memory  recall,  or  other  higher-order 
processes  that  the  operator  must  call  upon  during 
performance. 


Levels  for  Three  Cockpit  Concepts 


The  disadvantages  of  the  metric  are  Its  lack 
of  sensitivity  to  transient  effects  such  as  fatigue 
and  stress#  and  Its  rigidity  In  modeling  the  time 
sequence  of  events  In  the  mission  scenario.  This 
latter  problem  may  be  addressed  through 
Incorporating  a  method  of  "windowing"  the  period  of 
time  that  each  event  may  occur#  for  those  tasks 
that  are  likely  to  shift  their  start  and  stop  times 
In  the  mission.  A  "mean"  and  "standard  error" 
concept  could  be  applied  to  the  data  to  Introduce  a 
stochastic  component  Into  the  model  and  allow 
events  to  "occur*  with  an  element  of  randomness. 

A  validity  check  on  the  predictive  power  of 
W/INDEX  Is  the  most  critical  need  for  further 
metric  development.  We  are  planning  Initial 
validations  against  simulation  performance  In 
armored  and  rotorwing  crewstatlons  during  1965. 
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TABLE 

4.  1 

CONFLICT  MATRIX  OF  TIMESHARING  DIFFICULTY 

TASK 

WIN 

HMD 

CRT 

AUO 

STK 

key 

PIS 

Window  (WIN) 

.7 

Hal  mat  Display  (WC) 

.3 

.3 

Cockpit  CRT  (CRT) 

•  .9 

.3 

.5 

Auditory  Input  (AUO) 

.3 

.3 

.3 

.9 

Stick  Control  (STK) 

.1 

.1 

.1 

.1 

.9 

Kayboard  (KEY) 

.1 

.1 

.1 

.1 

.7 

.9 

Olscrata  Switch  (DIS) 

.1 

.1 

.1 

•  1 

.5 

a7 

a  7 

Spaach  (SPE) 

.1 

.1 

.1 

.3 

.3 

•5 

.1 

Cognltlva  (COG) 

.5 

.5 

.5 

.7 

.3 

•5 

al 

'vlvlv'’ 


CONFLICT  MATRIX  BETWEEN  CHANNELS 
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Typical  Interface/Activity 
Matrix,  Battle  Command  SCAT 

Honeywell 


Activity  Name 

VOC 

VIC 

VHC 

AUD 

MCO 

MDC 

S  PC 

COG 

FC 

Bob  Up 

1 

0 

1 

0 

3 

0 

0 

0 

FC 

Hover  in  ground  effect 

3 

0 

1 

0 

3 

0 

0 

0 

FC 

Hover  out  of  ground  effect 

1 

0 

2 

0 

3 

0 

0 

0 

FC 

Nap  of  the  earth 

3 

0 

2 

0 

5 

0 

0 

0 

FC 

Remask 

3 

0 

2 

0 

5 

0 

0 

0 

FC 

Select  flight  symbology 

0 

0 

0 

0 

0 

1 

0 

0 

NV 

Check  navigation 

0 

2 

0 

0 

0 

0 

0 

0 

NV 

Update  navigation 

0 

2 

0 

0 

0 

2 

0 

0 

CO 

Acknowledge  receipt  of  data 

0 

0 

0 

1 

0 

0 

0 

0 

CO 

Data  receive 

0 

0 

0 

2 

0 

0 

0 

0 

CO 

Data  transmit 

0 

0 

0 

0 

0 

0 

1 

0 

CO 

Transmit  battle  position  to  team 

0 

0 

0 

0 

0 

0 

2 

0 

ASE 

Assess  threats 

0 

0 

0 

0 

0 

0 

0 

3 

ASE 

Threat  evasive  maneuver 

0 

0 

0 

0 

0 

0 

0 

3 

ASE 

Threat  response  plan 

0 

0 

0 

0 

0 

0 

0 

4 

TA 

Acknowledge  eye-sensor  link 

0 

0 

1 

0 

0 

0 

0 

0 

TA 

Acknowledge  sensor  mode 

0 

1 

0 

0 

0 

0 

0 

0 

TA 

Acknowledge  sensor  ready 

0 

1 

0 

0 

0 

0 

0 

0 

TA 

Adjust  sensor 

0 

0 

1 

0 

1 

0 

0 

0 

TA 

Link  eye/sensor 

0 

0 

0 

0 

0 

1 

0 

0 

TA 

Select  sensor 

0 

1 

0 

0 

0 

2 

0 

0 

TA 

Select  sensor  mode 

0 

1 

0 

0 

0 

2 

0 

0 

TA 

Slew  sensor 

1 

0 

1 

0 

3 

0 

0 

0 

TA 

Target  search 

3 

0 

1 

0 

0 

0 

0 

0 

FRC 

Acknowledge  weapons  elect 

0 

0 

1 

0 

0 

0 

0 

0 

FRC 

Select  weapon 

0 

0 

0 

0 

0 

2 

0 

0 

MM 

Sensor  management 

0 

0 

0 

0 

0 

0 

0 

2 

MM 

Stores  management 

0 

0 

0 

0 

0 

0 

0 

3 

MM 

System  monitor 

0 

0 

0 

0 

0 

0 

0 

2 

MM 

Tactical  coordination 

0 

0 

0 

0 

0 

0 

0 

3 

MM 

Tactical  decision 

0 

0 

0 

0 

0 

0 

0 

2 
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INSTANTANEOUS  WORKLOAD  DEMAND  CALCULATION 
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Four  Workload  Estimate 
Calculation  Examples  Honeywell 


WORKLOAD  ANALYSIS 
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ATTENDANCE  LIST 


Mr.  Homer  Adkins 
CNET 

Code  NS  11 
Naval  Air  Station 
Pensacola,  FL  32508 
Av  922-4201 

Mr.  Robert  Bateman 

Boeing  Military  Airplane  Co. 

MS  K  76-23 
Wichita,  KS  67210 
(316)  526-2121 

Mr.  D.  Beevis 
uCIEM 

P.C.  Box  2000 
Downsview,  Ontario 
CANADA  M3M3B9 
AV  827-4101 

Dr.  Mike  Biferno 
Mail  Code  35-36 
Douglas  Aircraft  Co. 

3855  Lakewood  Blvd. 

Long  Beach,  CA  90840 
(213)  593-6645 

Mr.  harry  Britten-Austin 
Squadron  Leader 
AFWAL/FIGR 

Wright-Patterson  AFB,  OH  45433 
(513)  255-8259 

Dr.  Nancy  M.  Bucher 
Research  Manager 

Crew  Station  Research  and  Development  Office 
TA-8 

NASA-Anes  Research  Center 
Moffett  Field,  CA  94035 
(415)  694-4024 

Mr.  Larry  Butterbaugh 
AFWAL/FIGR 

Wright-Patterson  AFB,  Oh  45433 
(513)  255-8260 

Mr.  Ron  Camnarata 

Director 

USALABCOM 

Human  Engineering  Laboratory 
Attn:  SL  CHE -CC ( Cammarata) 

Aberdeen  Proving  Ground,  MD  21005-5001 
(301)  278-5950 
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Mr.  Dave  Carter 
US  Army  ARL  Commander 
Box  577 

SGRD-UMB-CS/Carter 
Fort  Rucker,  AL  36362 
Av  558-6862 

Mr.  Richara  E.  Christ 
Chief 

US  Army  Research  Institute  for  the  Behavioral 
and  Social  Sciences-Ft.  Bliss  Field  Unit 
ATT N :  Dr.  Christ 
P.0.  Box  6057 

Fort  Bliss,  TX  79506-0057 
(915)  568-4491 

Mr.  Larry  Clapp 
Code  918 

Naval  Avionics  Center 
6000  E.  21st  St. 

Indianapolis,  IN  46219-2189 
(317)  351-4476 

Eric  Crawford 
c/o  Dr.  Schiffler 
ASU/ENECh 

Wright-Patterson  AFB,  OH  45433 
Mr.  Pat  Dowd 

Crew  Technology  Division 
USAFSAM/VN 

Brooks  AFb,  TX  78235-5000 
AV  240-3464 

Dr.  Douglas  Eddy 

Crew  Technology  Division 

USAFSAM/VN 

Brooks  AFB,  TX  78232-5000 
AV  240-3464 

Ms.  DeAun  Faulkner 
Commander 

Naval  Air  Test  Center  (SX726) 

ATTN:  DeAun  Faulkner 
Patuxent  River,  MD  20670-5304 
AV  356-4157 

Major  Lucky  Goebel 
B-1B  FOT  &  E  ITDD 
Dyess  AFB,  TX  79607-5270 
AV  446-3514 

Mr.  Bruce  Hamilton 
78  Oxbow  Lane 
Northforci,  CT  06472 
(203)  386-5834 
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Dr.  Peter  Hancock 
University  of  Southern  California 
Department  of  Safety  Science 
ISSN,  USC 

Los  Angeles,  CA  90089-0021 
(213)  743-2S38 

Dr.  Randall  L.  harris,  Sr. 

Langley  Research  Center 
MS  1 52E 

hampton,  VA  23665-5225 
(804)  865-3S17 

Mr.  K.  C.  Hendy 
DC  I  EM 

P.0.  Box  2000 
Downsview,  Ontario 
CANADA  M3M3B9 
AV  827-4101 

Mr.  Dick  Horst 
ARD  Corp. 

5457  Twin  Knolls  Rd. 

Columbia,  MD  21045 
(301)  596-5845 

Dr.  Daniel  Jones 
USNRC 

DHFT/HF IB (AR 5209) 

Washington,  DC  20555 
(301)  492-4846 

Ms.  Karen  Jones 
Coast  Guard  Institute 
US  Coast  Guard 
P.0.  Sub  Station  18 
Oklahoma  City,  OK  7  3169 
(405)  686-2416 
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Mr.  Jerry  Kennebrew 
580  N.  Park  St.  flO 
Columbus,  OH  43215 
(614)  860-7193 

Ms.  June  E.  Lay 
General  Dynamics 
P.0.  Box  748 
MZ  1766 

Ft.  Worth,  TX  76101 
(817)  763-2780 

Mr.  Kwan  Lee 


..•V. 


3128  Ceba  Dr. 

Louisiana  State  University 
Baton  Rouge,  LA  70803 
(504)  388-5369 


Dr.  Nit  a  Lev/ is 

Crew  Technology  Division 

USAFSAM/VN 

brooks  AFb,  TX  78235-5000 
AV  240-3464 

Ms.  Gretchen  Lizza 
c/o  Dr.  Schiffler 
ASD/ENECh 

Wright-Patterson  AFB,  OH  45433 
(513)  237-0473 

Mr.  John  Lockett 

Director 

bSALABCOM 

human  Engineering  Laboratory 
Attn:  SLCHE-CC( Lockett) 

Aberdeen  Proving  Ground,  MD  21005-5001 
(301)  278-5950 

Mr.  Daniel  McCrobie 
MZ  601-19 
General  Dynamics 
P.0.  Box  50-800 
Ontario,  CA  91761 
(714)  945-7562 

Mr.  Lou  fleyers 
battel le 
505  King  Avenue 
Columbus,  OH  43201 
(614)  424-7166 

Dr.  uames  Miller 

Crew  Technology  Division 

USAFSAM/VN 

brooks  AFB,  TX  78235-5000 
AV  240-3464 

Ms.  Anarea  Morris 
boeing  VERT0L 
P.0.  Box  16858 
MS-P3032 

Philadelphia,  PA  19142 
(215)  499-9635 

Ms.  Deborah  Mumnaw 
6520  TESTG/ENAH 
Edwards  AFb,  CA  93523-5000 
(805)  277-3334 

LT  COL  Mark  Nataupsky 
MS  1 52 -E 

NASA-Langley  Research  Center 
hampton,  VA  23665-5225 
(804)  865-3917 


[■is.  Laurie  Quill 
Beville  Engineering,  Inc. 

270  Regency  Ridge,  Suite  214 
Dayton,  OH  45459 
(513)  434-1093 

Dr.  Matilda  Reeder 
Army  Research  Institute 
PER  I - IR 

Ft.  Rucker,  AL  36362-5000 
AV  558-4204 

Dr.  Richard  J.  Schiffler 
ASD/ENECH 

Wright-Patterson  AFB,  OH  45433 
(513)  255-5597 

CAPT  John  K.  Schmidt 

Director 

USALA6C0M 

Human  Engineering  Laboratory 
Attn:  SLCHE-CC(CAPT  Schmidt) 

Aberdeen  Proving  Ground,  MD  21005-5001 
(301)  278-5836 

Dr.  Larry  Shadow 
6545  Test  Group/TIE 
Hill  AFb,  UT  84056-5000 
AV  458-6195 

Ms.  Kirn  Siler 
414  E.  Cedar,  Apt.  3 
Burbank,  CA  91501 
(818)  847-7591 

Mr.  Rom  Simmons 
US  Army  ARL  Commander 
Box  577 

SGRD-UAB-CS/Carter 
Fort  Rucker,  AL  36362 
AV  558-6862 

Ms.  Mary  Lou  Smith 
c/o  Dr.  Schiffler 
ASD/ENECH 

Wright-Patterson  AFB,  OH  45433 
(51  3)  429-2252 

Mr.  Paul  Stager 
Department  of  Psychology 
York  University 
4700  Keele  Street 
Toronto,  Ontario 
CANADA  M3J1P3 
(416)  736-2100 
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Mr.  Lewis  Stone 
bS  Army  ARL  Comnanaer 
Box  577 

SGRD-UAb-CS/Stone 
Fort  Rucker,  AL  36362 
AV  558-6853 


Mr.  Richard  Tauson 
Director 

USA  human  Engineering  Laboratory 
ATTN:  AMXHE-FT  (R.  A.  Tauson) 

Aberdeen  Proving  Ground,  MD  21005-5001 
AV  298-5874 


Mr.  Marvin  Thandsen 
c/o  Dr.  Schiffler 
ASD/ENECH 

Wright -Patterson  AFB,  OH  45433 
(513)  767-2691 


Ms.  Mary  Thiessen 
General  Dynamics 
P.0.  Box  748 
MZ  1766 

Ft.  Worth,  TX  76101 
(817)  763-2780 


Ms.  Maria  L.  Thomas 

Walter  Reed  Army  Institute  of  Research 
Division  of  Neuropsychiatry 
Department  of  Human  Behavioral  Biology 
ATTN:  SGRD-UWI-C 
Washington,  DC  20307-5100 
(301)  427-5521 


Ms.  Pamela  Tsang 
MS  239-3 

NASA-Anes  Research  Center 
Moffett  Field,  CA  94035 
(415)  694-6187 


Mr.  Michael  Vidulich 
MS  239-3 

NASA-Ames  Research  Center 
Moffett  Field,  CA  94035 
(415)  694-6187 


Mr.  Bob  Weneck 
Eastman  Kodak  Co. 
Ergonomics  Group 
Bldg  320,  2nd  Floor 
Kodak  Park 

Rochester,  NY  14650 
(716)  722-6973 
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